diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 00000000..d893b45a --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,7 @@ +include requirements.txt +include LICENSE +include README.md +prune test/ +prune reproduction/ +prune fastNLP/api +prune fastNLP/automl \ No newline at end of file diff --git a/README.md b/README.md index 5346fbd7..9d949482 100644 --- a/README.md +++ b/README.md @@ -2,88 +2,114 @@ [![Build Status](https://travis-ci.org/fastnlp/fastNLP.svg?branch=master)](https://travis-ci.org/fastnlp/fastNLP) [![codecov](https://codecov.io/gh/fastnlp/fastNLP/branch/master/graph/badge.svg)](https://codecov.io/gh/fastnlp/fastNLP) -[![PyPI version](https://badge.fury.io/py/fastNLP.svg)](https://badge.fury.io/py/fastNLP) +[![Pypi](https://img.shields.io/pypi/v/fastNLP.svg)](https://pypi.org/project/fastNLP) ![Hex.pm](https://img.shields.io/hexpm/l/plug.svg) [![Documentation Status](https://readthedocs.org/projects/fastnlp/badge/?version=latest)](http://fastnlp.readthedocs.io/?badge=latest) -FastNLP is a modular Natural Language Processing system based on PyTorch, built for fast development of NLP models. +fastNLP 是一款轻量级的 NLP 处理套件。你既可以使用它快速地完成一个命名实体识别（NER）、中文分词或文本分类任务； 也可以使用他构建许多复杂的网络模型，进行科研。它具有如下的特性： + +- 统一的Tabular式数据容器，让数据预处理过程简洁明了。内置多种数据集的DataSet Loader，省去预处理代码。 +- 各种方便的NLP工具，例如预处理embedding加载; 中间数据cache等; +- 详尽的中文文档以供查阅； +- 提供诸多高级模块，例如Variational LSTM, Transformer, CRF等; +- 封装CNNText，Biaffine等模型可供直接使用; +- 便捷且具有扩展性的训练器; 提供多种内置callback函数，方便实验记录、异常捕获等。 + + +## 安装指南 + +fastNLP 依赖如下包: + ++ numpy ++ torch>=0.4.0 ++ tqdm ++ nltk + +其中torch的安装可能与操作系统及 CUDA 的版本相关，请参见 PyTorch 官网 。 +在依赖包安装完成的情况，您可以在命令行执行如下指令完成安装 + +```shell +pip install fastNLP +``` + + +## 参考资源 + +- [文档](https://fastnlp.readthedocs.io/zh/latest/) +- [源码](https://github.com/fastnlp/fastNLP) + + + +## 内置组件 + +大部分用于的 NLP 任务神经网络都可以看做由编码（encoder）、聚合（aggregator）、解码（decoder）三种模块组成。 + + +![](./docs/source/figures/text_classification.png) + +fastNLP 在 modules 模块中内置了三种模块的诸多组件，可以帮助用户快速搭建自己所需的网络。 三种模块的功能和常见组件如下: -A deep learning NLP model is the composition of three types of modules: - - - + + + - + - + - +
module type functionality example 类型 功能 例子
encoder encode the input into some abstract representation 将输入编码为具有具 有表示能力的向量 embedding, RNN, CNN, transformer
aggregator aggregate and reduce information 从多个向量中聚合信息 self-attention, max-pooling
decoder decode the representation into the output 将具有某种表示意义的 向量解码为需要的输出 形式 MLP, CRF
-For example: - -![](docs/source/figures/text_classification.png) - -## Requirements - -- Python>=3.6 -- numpy>=1.14.2 -- torch>=0.4.0 -- tensorboardX -- tqdm>=4.28.1 - - -## Resources -- [Tutorials](https://github.com/fastnlp/fastNLP/tree/master/tutorials) -- [Documentation](https://fastnlp.readthedocs.io/en/latest/) -- [Source Code](https://github.com/fastnlp/fastNLP) +## 完整模型 +fastNLP 为不同的 NLP 任务实现了许多完整的模型，它们都经过了训练和测试。 +你可以在以下两个地方查看相关信息 +- [介绍](reproduction/) +- [源码](fastNLP/models/) -## Installation -Run the following commands to install fastNLP package. -```shell -pip install fastNLP -``` +## 项目结构 +![](./docs/source/figures/workflow.png) -## Project Structure +fastNLP的大致工作流程如上图所示，而项目结构如下： - - - - - + - + - + - + - +
fastNLP an open-source NLP library
fastNLP.api APIs for end-to-end prediction 开源的自然语言处理库
fastNLP.core data representation & train/test procedure 实现了核心功能，包括数据处理组件、训练器、测速器等
fastNLP.models a collection of NLP models 实现了一些完整的神经网络模型
fastNLP.modules a collection of PyTorch sub-models/components/wheels 实现了用于搭建神经网络模型的诸多组件
fastNLP.io readers & savers 实现了读写功能，包括数据读入，模型读写等
+ + +
+ +*In memory of @FengZiYjun. May his soul rest in peace. We will miss you very very much!* diff --git a/codecov.yml b/codecov.yml new file mode 100644 index 00000000..f91e0445 --- /dev/null +++ b/codecov.yml @@ -0,0 +1,5 @@ +ignore: +- "reproduction" # ignore folders and all its contents +- "setup.py" +- "docs" +- "tutorials" \ No newline at end of file diff --git a/docs/Makefile b/docs/Makefile index e978dfe6..6ba2fa54 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -3,6 +3,7 @@ # You can set these variables from the command line. SPHINXOPTS = +SPHINXAPIDOC = sphinx-apidoc SPHINXBUILD = sphinx-build SPHINXPROJ = fastNLP SOURCEDIR = source @@ -12,6 +13,12 @@ BUILDDIR = build help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) +apidoc: + $(SPHINXAPIDOC) -efM -o source ../$(SPHINXPROJ) + +server: + cd build/html && python -m http.server + .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new diff --git a/docs/source/conf.py b/docs/source/conf.py index e449a9f8..3e9753af 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -14,6 +14,7 @@ # import os import sys + sys.path.insert(0, os.path.abspath('../../')) # -- Project information ----------------------------------------------------- @@ -23,10 +24,9 @@ copyright = '2018, xpqiu' author = 'xpqiu' # The short X.Y version -version = '0.2' +version = '0.4' # The full version, including alpha/beta/rc tags -release = '0.2' - +release = '0.4' # -- General configuration --------------------------------------------------- @@ -42,9 +42,15 @@ extensions = [ 'sphinx.ext.viewcode', 'sphinx.ext.autosummary', 'sphinx.ext.mathjax', - + 'sphinx.ext.todo' ] +autodoc_default_options = { + 'member-order': 'bysource', + 'special-members': '__init__', + 'undoc-members': True, +} + # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] @@ -62,17 +68,16 @@ master_doc = 'index' # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. -language = None +language = "zh_CN" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path . -exclude_patterns = [] +exclude_patterns = ['modules.rst'] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' - # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for @@ -84,7 +89,10 @@ html_theme = 'sphinx_rtd_theme' # further. For a list of options available for each theme, see the # documentation. # -# html_theme_options = {} +html_theme_options = { + 'collapse_navigation': False, + 'titles_only': True +} # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, @@ -107,22 +115,21 @@ html_static_path = ['_static'] # Output file base name for HTML help builder. htmlhelp_basename = 'fastNLPdoc' - # -- Options for LaTeX output ------------------------------------------------ latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', - + # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', - + # Additional stuff for the LaTeX preamble. # # 'preamble': '', - + # Latex figure (float) alignment # # 'figure_align': 'htbp', @@ -136,7 +143,6 @@ latex_documents = [ 'xpqiu', 'manual'), ] - # -- Options for manual page output ------------------------------------------ # One entry per manual page. List of tuples @@ -146,7 +152,6 @@ man_pages = [ [author], 1) ] - # -- Options for Texinfo output ---------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples @@ -159,4 +164,14 @@ texinfo_documents = [ ] -# -- Extension configuration ------------------------------------------------- \ No newline at end of file +# -- Extension configuration ------------------------------------------------- +def maybe_skip_member(app, what, name, obj, skip, options): + if name.startswith("_"): + return True + if obj.__doc__ is None: + return True + return False + + +def setup(app): + app.connect('autodoc-skip-member', maybe_skip_member) diff --git a/docs/source/fastNLP.api.rst b/docs/source/fastNLP.api.rst deleted file mode 100644 index eb9192da..00000000 --- a/docs/source/fastNLP.api.rst +++ /dev/null @@ -1,36 +0,0 @@ -fastNLP.api -============ - -fastNLP.api.api ----------------- - -.. automodule:: fastNLP.api.api - :members: - -fastNLP.api.converter ----------------------- - -.. automodule:: fastNLP.api.converter - :members: - -fastNLP.api.model\_zoo ------------------------ - -.. automodule:: fastNLP.api.model_zoo - :members: - -fastNLP.api.pipeline ---------------------- - -.. automodule:: fastNLP.api.pipeline - :members: - -fastNLP.api.processor ----------------------- - -.. automodule:: fastNLP.api.processor - :members: - - -.. automodule:: fastNLP.api - :members: diff --git a/docs/source/fastNLP.core.batch.rst b/docs/source/fastNLP.core.batch.rst new file mode 100644 index 00000000..33a5b730 --- /dev/null +++ b/docs/source/fastNLP.core.batch.rst @@ -0,0 +1,7 @@ +fastNLP.core.batch +================== + +.. automodule:: fastNLP.core.batch + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.callback.rst b/docs/source/fastNLP.core.callback.rst new file mode 100644 index 00000000..31ec627b --- /dev/null +++ b/docs/source/fastNLP.core.callback.rst @@ -0,0 +1,7 @@ +fastNLP.core.callback +===================== + +.. automodule:: fastNLP.core.callback + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.const.rst b/docs/source/fastNLP.core.const.rst new file mode 100644 index 00000000..c9e3bd97 --- /dev/null +++ b/docs/source/fastNLP.core.const.rst @@ -0,0 +1,7 @@ +fastNLP.core.const +================== + +.. automodule:: fastNLP.core.const + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.dataset.rst b/docs/source/fastNLP.core.dataset.rst new file mode 100644 index 00000000..b377cb0f --- /dev/null +++ b/docs/source/fastNLP.core.dataset.rst @@ -0,0 +1,7 @@ +fastNLP.core.dataset +==================== + +.. automodule:: fastNLP.core.dataset + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.field.rst b/docs/source/fastNLP.core.field.rst new file mode 100644 index 00000000..7686e79a --- /dev/null +++ b/docs/source/fastNLP.core.field.rst @@ -0,0 +1,7 @@ +fastNLP.core.field +================== + +.. automodule:: fastNLP.core.field + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.instance.rst b/docs/source/fastNLP.core.instance.rst new file mode 100644 index 00000000..14393a91 --- /dev/null +++ b/docs/source/fastNLP.core.instance.rst @@ -0,0 +1,7 @@ +fastNLP.core.instance +===================== + +.. automodule:: fastNLP.core.instance + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.losses.rst b/docs/source/fastNLP.core.losses.rst new file mode 100644 index 00000000..d2dd492b --- /dev/null +++ b/docs/source/fastNLP.core.losses.rst @@ -0,0 +1,7 @@ +fastNLP.core.losses +=================== + +.. automodule:: fastNLP.core.losses + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.metrics.rst b/docs/source/fastNLP.core.metrics.rst new file mode 100644 index 00000000..69afff36 --- /dev/null +++ b/docs/source/fastNLP.core.metrics.rst @@ -0,0 +1,7 @@ +fastNLP.core.metrics +==================== + +.. automodule:: fastNLP.core.metrics + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.optimizer.rst b/docs/source/fastNLP.core.optimizer.rst new file mode 100644 index 00000000..e2100d2e --- /dev/null +++ b/docs/source/fastNLP.core.optimizer.rst @@ -0,0 +1,7 @@ +fastNLP.core.optimizer +====================== + +.. automodule:: fastNLP.core.optimizer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.rst b/docs/source/fastNLP.core.rst index b9f6c89f..82c13e46 100644 --- a/docs/source/fastNLP.core.rst +++ b/docs/source/fastNLP.core.rst @@ -1,84 +1,29 @@ -fastNLP.core -============= - -fastNLP.core.batch -------------------- - -.. automodule:: fastNLP.core.batch - :members: - -fastNLP.core.dataset ---------------------- - -.. automodule:: fastNLP.core.dataset - :members: - -fastNLP.core.fieldarray ------------------------- - -.. automodule:: fastNLP.core.fieldarray - :members: - -fastNLP.core.instance ----------------------- - -.. automodule:: fastNLP.core.instance - :members: - -fastNLP.core.losses --------------------- - -.. automodule:: fastNLP.core.losses - :members: - -fastNLP.core.metrics ---------------------- - -.. automodule:: fastNLP.core.metrics - :members: - -fastNLP.core.optimizer ------------------------ - -.. automodule:: fastNLP.core.optimizer - :members: - -fastNLP.core.predictor ------------------------ - -.. automodule:: fastNLP.core.predictor - :members: - -fastNLP.core.sampler ---------------------- - -.. automodule:: fastNLP.core.sampler - :members: - -fastNLP.core.tester --------------------- - -.. automodule:: fastNLP.core.tester - :members: - -fastNLP.core.trainer ---------------------- - -.. automodule:: fastNLP.core.trainer - :members: - -fastNLP.core.utils -------------------- - -.. automodule:: fastNLP.core.utils - :members: - -fastNLP.core.vocabulary ------------------------- - -.. automodule:: fastNLP.core.vocabulary - :members: - +fastNLP.core +============ .. automodule:: fastNLP.core :members: + :undoc-members: + :show-inheritance: + +子模块 +---------- + +.. toctree:: + :titlesonly: + + fastNLP.core.batch + fastNLP.core.callback + fastNLP.core.const + fastNLP.core.dataset + fastNLP.core.field + fastNLP.core.instance + fastNLP.core.losses + fastNLP.core.metrics + fastNLP.core.optimizer + fastNLP.core.sampler + fastNLP.core.tester + fastNLP.core.trainer + fastNLP.core.utils + fastNLP.core.vocabulary + diff --git a/docs/source/fastNLP.core.sampler.rst b/docs/source/fastNLP.core.sampler.rst new file mode 100644 index 00000000..1810d59c --- /dev/null +++ b/docs/source/fastNLP.core.sampler.rst @@ -0,0 +1,7 @@ +fastNLP.core.sampler +==================== + +.. automodule:: fastNLP.core.sampler + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.tester.rst b/docs/source/fastNLP.core.tester.rst new file mode 100644 index 00000000..a9e7e09f --- /dev/null +++ b/docs/source/fastNLP.core.tester.rst @@ -0,0 +1,7 @@ +fastNLP.core.tester +=================== + +.. automodule:: fastNLP.core.tester + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.trainer.rst b/docs/source/fastNLP.core.trainer.rst new file mode 100644 index 00000000..9e518d4b --- /dev/null +++ b/docs/source/fastNLP.core.trainer.rst @@ -0,0 +1,7 @@ +fastNLP.core.trainer +==================== + +.. automodule:: fastNLP.core.trainer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.utils.rst b/docs/source/fastNLP.core.utils.rst new file mode 100644 index 00000000..fcd3f50c --- /dev/null +++ b/docs/source/fastNLP.core.utils.rst @@ -0,0 +1,7 @@ +fastNLP.core.utils +================== + +.. automodule:: fastNLP.core.utils + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.core.vocabulary.rst b/docs/source/fastNLP.core.vocabulary.rst new file mode 100644 index 00000000..b3bf4bac --- /dev/null +++ b/docs/source/fastNLP.core.vocabulary.rst @@ -0,0 +1,7 @@ +fastNLP.core.vocabulary +======================= + +.. automodule:: fastNLP.core.vocabulary + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.io.base_loader.rst b/docs/source/fastNLP.io.base_loader.rst new file mode 100644 index 00000000..c1f9ac14 --- /dev/null +++ b/docs/source/fastNLP.io.base_loader.rst @@ -0,0 +1,7 @@ +fastNLP.io.base\_loader +======================= + +.. automodule:: fastNLP.io.base_loader + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.io.dataset_loader.rst b/docs/source/fastNLP.io.dataset_loader.rst new file mode 100644 index 00000000..d6663e59 --- /dev/null +++ b/docs/source/fastNLP.io.dataset_loader.rst @@ -0,0 +1,7 @@ +fastNLP.io.dataset\_loader +========================== + +.. automodule:: fastNLP.io.dataset_loader + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.io.embed_loader.rst b/docs/source/fastNLP.io.embed_loader.rst new file mode 100644 index 00000000..7a8e730c --- /dev/null +++ b/docs/source/fastNLP.io.embed_loader.rst @@ -0,0 +1,7 @@ +fastNLP.io.embed\_loader +======================== + +.. automodule:: fastNLP.io.embed_loader + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.io.model_io.rst b/docs/source/fastNLP.io.model_io.rst new file mode 100644 index 00000000..50d4c25a --- /dev/null +++ b/docs/source/fastNLP.io.model_io.rst @@ -0,0 +1,7 @@ +fastNLP.io.model\_io +==================== + +.. automodule:: fastNLP.io.model_io + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.io.rst b/docs/source/fastNLP.io.rst index d91e0d1c..fad05a21 100644 --- a/docs/source/fastNLP.io.rst +++ b/docs/source/fastNLP.io.rst @@ -1,42 +1,19 @@ -fastNLP.io -=========== +fastNLP.io +========== -fastNLP.io.base\_loader ------------------------- - -.. automodule:: fastNLP.io.base_loader - :members: - -fastNLP.io.config\_io ----------------------- - -.. automodule:: fastNLP.io.config_io - :members: - -fastNLP.io.dataset\_loader ---------------------------- - -.. automodule:: fastNLP.io.dataset_loader - :members: - -fastNLP.io.embed\_loader -------------------------- - -.. automodule:: fastNLP.io.embed_loader - :members: - -fastNLP.io.logger ------------------- - -.. automodule:: fastNLP.io.logger +.. automodule:: fastNLP.io :members: + :undoc-members: + :show-inheritance: -fastNLP.io.model\_io ---------------------- +子模块 +---------- -.. automodule:: fastNLP.io.model_io - :members: +.. toctree:: + :titlesonly: + fastNLP.io.base_loader + fastNLP.io.dataset_loader + fastNLP.io.embed_loader + fastNLP.io.model_io -.. automodule:: fastNLP.io - :members: diff --git a/docs/source/fastNLP.models.biaffine_parser.rst b/docs/source/fastNLP.models.biaffine_parser.rst new file mode 100644 index 00000000..a3dd1836 --- /dev/null +++ b/docs/source/fastNLP.models.biaffine_parser.rst @@ -0,0 +1,7 @@ +fastNLP.models.biaffine\_parser +=============================== + +.. automodule:: fastNLP.models.biaffine_parser + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.models.cnn_text_classification.rst b/docs/source/fastNLP.models.cnn_text_classification.rst new file mode 100644 index 00000000..a935d0bf --- /dev/null +++ b/docs/source/fastNLP.models.cnn_text_classification.rst @@ -0,0 +1,7 @@ +fastNLP.models.cnn\_text\_classification +======================================== + +.. automodule:: fastNLP.models.cnn_text_classification + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.models.rst b/docs/source/fastNLP.models.rst index 7452fdf6..5858ebcd 100644 --- a/docs/source/fastNLP.models.rst +++ b/docs/source/fastNLP.models.rst @@ -1,42 +1,20 @@ -fastNLP.models -=============== +fastNLP.models +============== -fastNLP.models.base\_model ---------------------------- - -.. automodule:: fastNLP.models.base_model - :members: - -fastNLP.models.biaffine\_parser --------------------------------- - -.. automodule:: fastNLP.models.biaffine_parser - :members: - -fastNLP.models.char\_language\_model -------------------------------------- - -.. automodule:: fastNLP.models.char_language_model - :members: - -fastNLP.models.cnn\_text\_classification ------------------------------------------ - -.. automodule:: fastNLP.models.cnn_text_classification - :members: - -fastNLP.models.sequence\_modeling ----------------------------------- - -.. automodule:: fastNLP.models.sequence_modeling +.. automodule:: fastNLP.models :members: + :undoc-members: + :show-inheritance: -fastNLP.models.snli --------------------- +子模块 +---------- -.. automodule:: fastNLP.models.snli - :members: +.. toctree:: + :titlesonly: + fastNLP.models.biaffine_parser + fastNLP.models.cnn_text_classification + fastNLP.models.sequence_labeling + fastNLP.models.snli + fastNLP.models.star_transformer -.. automodule:: fastNLP.models - :members: diff --git a/docs/source/fastNLP.models.sequence_labeling.rst b/docs/source/fastNLP.models.sequence_labeling.rst new file mode 100644 index 00000000..6d569fe1 --- /dev/null +++ b/docs/source/fastNLP.models.sequence_labeling.rst @@ -0,0 +1,7 @@ +fastNLP.models.sequence\_labeling +================================= + +.. automodule:: fastNLP.models.sequence_labeling + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.models.snli.rst b/docs/source/fastNLP.models.snli.rst new file mode 100644 index 00000000..24c2cc53 --- /dev/null +++ b/docs/source/fastNLP.models.snli.rst @@ -0,0 +1,7 @@ +fastNLP.models.snli +=================== + +.. automodule:: fastNLP.models.snli + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.models.star_transformer.rst b/docs/source/fastNLP.models.star_transformer.rst new file mode 100644 index 00000000..c93fb8cd --- /dev/null +++ b/docs/source/fastNLP.models.star_transformer.rst @@ -0,0 +1,7 @@ +fastNLP.models.star\_transformer +================================ + +.. automodule:: fastNLP.models.star_transformer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.aggregator.attention.rst b/docs/source/fastNLP.modules.aggregator.attention.rst new file mode 100644 index 00000000..dc9c2b53 --- /dev/null +++ b/docs/source/fastNLP.modules.aggregator.attention.rst @@ -0,0 +1,7 @@ +fastNLP.modules.aggregator.attention +==================================== + +.. automodule:: fastNLP.modules.aggregator.attention + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.aggregator.pooling.rst b/docs/source/fastNLP.modules.aggregator.pooling.rst new file mode 100644 index 00000000..162f889d --- /dev/null +++ b/docs/source/fastNLP.modules.aggregator.pooling.rst @@ -0,0 +1,7 @@ +fastNLP.modules.aggregator.pooling +================================== + +.. automodule:: fastNLP.modules.aggregator.pooling + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.aggregator.rst b/docs/source/fastNLP.modules.aggregator.rst index 073da4a5..44398325 100644 --- a/docs/source/fastNLP.modules.aggregator.rst +++ b/docs/source/fastNLP.modules.aggregator.rst @@ -1,36 +1,17 @@ -fastNLP.modules.aggregator -=========================== +fastNLP.modules.aggregator +========================== -fastNLP.modules.aggregator.attention -------------------------------------- - -.. automodule:: fastNLP.modules.aggregator.attention - :members: - -fastNLP.modules.aggregator.avg\_pool -------------------------------------- - -.. automodule:: fastNLP.modules.aggregator.avg_pool - :members: - -fastNLP.modules.aggregator.kmax\_pool --------------------------------------- - -.. automodule:: fastNLP.modules.aggregator.kmax_pool +.. automodule:: fastNLP.modules.aggregator :members: + :undoc-members: + :show-inheritance: -fastNLP.modules.aggregator.max\_pool -------------------------------------- - -.. automodule:: fastNLP.modules.aggregator.max_pool - :members: +子模块 +---------- -fastNLP.modules.aggregator.self\_attention -------------------------------------------- +.. toctree:: + :titlesonly: -.. automodule:: fastNLP.modules.aggregator.self_attention - :members: + fastNLP.modules.aggregator.attention + fastNLP.modules.aggregator.pooling - -.. automodule:: fastNLP.modules.aggregator - :members: diff --git a/docs/source/fastNLP.modules.decoder.crf.rst b/docs/source/fastNLP.modules.decoder.crf.rst new file mode 100644 index 00000000..6d5b0d5b --- /dev/null +++ b/docs/source/fastNLP.modules.decoder.crf.rst @@ -0,0 +1,7 @@ +fastNLP.modules.decoder.CRF +=========================== + +.. automodule:: fastNLP.modules.decoder.crf + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.decoder.mlp.rst b/docs/source/fastNLP.modules.decoder.mlp.rst new file mode 100644 index 00000000..7d661ebf --- /dev/null +++ b/docs/source/fastNLP.modules.decoder.mlp.rst @@ -0,0 +1,7 @@ +fastNLP.modules.decoder.MLP +=========================== + +.. automodule:: fastNLP.modules.decoder.mlp + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.decoder.rst b/docs/source/fastNLP.modules.decoder.rst index 6844543a..e42a9f39 100644 --- a/docs/source/fastNLP.modules.decoder.rst +++ b/docs/source/fastNLP.modules.decoder.rst @@ -1,18 +1,18 @@ -fastNLP.modules.decoder -======================== +fastNLP.modules.decoder +======================= -fastNLP.modules.decoder.CRF ----------------------------- - -.. automodule:: fastNLP.modules.decoder.CRF +.. automodule:: fastNLP.modules.decoder :members: + :undoc-members: + :show-inheritance: -fastNLP.modules.decoder.MLP ----------------------------- +子模块 +---------- -.. automodule:: fastNLP.modules.decoder.MLP - :members: +.. toctree:: + :titlesonly: + fastNLP.modules.decoder.crf + fastNLP.modules.decoder.mlp + fastNLP.modules.decoder.utils -.. automodule:: fastNLP.modules.decoder - :members: diff --git a/docs/source/fastNLP.modules.decoder.utils.rst b/docs/source/fastNLP.modules.decoder.utils.rst new file mode 100644 index 00000000..da979d99 --- /dev/null +++ b/docs/source/fastNLP.modules.decoder.utils.rst @@ -0,0 +1,7 @@ +fastNLP.modules.decoder.utils +============================= + +.. automodule:: fastNLP.modules.decoder.utils + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.bert.rst b/docs/source/fastNLP.modules.encoder.bert.rst new file mode 100644 index 00000000..66bd0bbd --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.bert.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.bert +============================ + +.. automodule:: fastNLP.modules.encoder.bert + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.char_encoder.rst b/docs/source/fastNLP.modules.encoder.char_encoder.rst new file mode 100644 index 00000000..61ea3340 --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.char_encoder.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.char\_encoder +===================================== + +.. automodule:: fastNLP.modules.encoder.char_encoder + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.conv_maxpool.rst b/docs/source/fastNLP.modules.encoder.conv_maxpool.rst new file mode 100644 index 00000000..7058a723 --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.conv_maxpool.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.conv\_maxpool +===================================== + +.. automodule:: fastNLP.modules.encoder.conv_maxpool + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.embedding.rst b/docs/source/fastNLP.modules.encoder.embedding.rst new file mode 100644 index 00000000..4427b3bf --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.embedding.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.embedding +================================= + +.. automodule:: fastNLP.modules.encoder.embedding + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.lstm.rst b/docs/source/fastNLP.modules.encoder.lstm.rst new file mode 100644 index 00000000..f9cbea88 --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.lstm.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.lstm +============================ + +.. automodule:: fastNLP.modules.encoder.lstm + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.rst b/docs/source/fastNLP.modules.encoder.rst index ea8fc699..b15232fa 100644 --- a/docs/source/fastNLP.modules.encoder.rst +++ b/docs/source/fastNLP.modules.encoder.rst @@ -1,60 +1,23 @@ -fastNLP.modules.encoder -======================== +fastNLP.modules.encoder +======================= -fastNLP.modules.encoder.char\_embedding ----------------------------------------- - -.. automodule:: fastNLP.modules.encoder.char_embedding - :members: - -fastNLP.modules.encoder.conv ------------------------------ - -.. automodule:: fastNLP.modules.encoder.conv - :members: - -fastNLP.modules.encoder.conv\_maxpool --------------------------------------- - -.. automodule:: fastNLP.modules.encoder.conv_maxpool - :members: - -fastNLP.modules.encoder.embedding ----------------------------------- - -.. automodule:: fastNLP.modules.encoder.embedding - :members: - -fastNLP.modules.encoder.linear -------------------------------- - -.. automodule:: fastNLP.modules.encoder.linear - :members: - -fastNLP.modules.encoder.lstm ------------------------------ - -.. automodule:: fastNLP.modules.encoder.lstm - :members: - -fastNLP.modules.encoder.masked\_rnn ------------------------------------- - -.. automodule:: fastNLP.modules.encoder.masked_rnn - :members: - -fastNLP.modules.encoder.transformer ------------------------------------- - -.. automodule:: fastNLP.modules.encoder.transformer +.. automodule:: fastNLP.modules.encoder :members: + :undoc-members: + :show-inheritance: -fastNLP.modules.encoder.variational\_rnn ------------------------------------------ +子模块 +---------- -.. automodule:: fastNLP.modules.encoder.variational_rnn - :members: +.. toctree:: + :titlesonly: + fastNLP.modules.encoder.bert + fastNLP.modules.encoder.char_encoder + fastNLP.modules.encoder.conv_maxpool + fastNLP.modules.encoder.embedding + fastNLP.modules.encoder.lstm + fastNLP.modules.encoder.star_transformer + fastNLP.modules.encoder.transformer + fastNLP.modules.encoder.variational_rnn -.. automodule:: fastNLP.modules.encoder - :members: diff --git a/docs/source/fastNLP.modules.encoder.star_transformer.rst b/docs/source/fastNLP.modules.encoder.star_transformer.rst new file mode 100644 index 00000000..0c406782 --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.star_transformer.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.star\_transformer +========================================= + +.. automodule:: fastNLP.modules.encoder.star_transformer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.transformer.rst b/docs/source/fastNLP.modules.encoder.transformer.rst new file mode 100644 index 00000000..6a40c597 --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.transformer.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.transformer +=================================== + +.. automodule:: fastNLP.modules.encoder.transformer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.encoder.variational_rnn.rst b/docs/source/fastNLP.modules.encoder.variational_rnn.rst new file mode 100644 index 00000000..348fb3d8 --- /dev/null +++ b/docs/source/fastNLP.modules.encoder.variational_rnn.rst @@ -0,0 +1,7 @@ +fastNLP.modules.encoder.variational\_rnn +======================================== + +.. automodule:: fastNLP.modules.encoder.variational_rnn + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/fastNLP.modules.rst b/docs/source/fastNLP.modules.rst index 965fb27d..d04ccdcf 100644 --- a/docs/source/fastNLP.modules.rst +++ b/docs/source/fastNLP.modules.rst @@ -1,30 +1,17 @@ -fastNLP.modules -================ +fastNLP.modules +=============== -.. toctree:: - - fastNLP.modules.aggregator - fastNLP.modules.decoder - fastNLP.modules.encoder - -fastNLP.modules.dropout ------------------------- - -.. automodule:: fastNLP.modules.dropout - :members: - -fastNLP.modules.other\_modules -------------------------------- - -.. automodule:: fastNLP.modules.other_modules +.. automodule:: fastNLP.modules :members: + :undoc-members: + :show-inheritance: -fastNLP.modules.utils ----------------------- - -.. automodule:: fastNLP.modules.utils - :members: +子模块 +----------- +.. toctree:: + :titlesonly: -.. automodule:: fastNLP.modules - :members: + fastNLP.modules.aggregator + fastNLP.modules.decoder + fastNLP.modules.encoder \ No newline at end of file diff --git a/docs/source/fastNLP.rst b/docs/source/fastNLP.rst index 61882359..f0c3d41c 100644 --- a/docs/source/fastNLP.rst +++ b/docs/source/fastNLP.rst @@ -1,13 +1,20 @@ -fastNLP -======== +API 文档 +=============== + +.. automodule:: fastNLP + :members: + :undoc-members: + :show-inheritance: + +内部模块 +----------- .. toctree:: + :titlesonly: + :maxdepth: 3 - fastNLP.api fastNLP.core fastNLP.io - fastNLP.models fastNLP.modules + fastNLP.models -.. automodule:: fastNLP - :members: diff --git a/docs/source/figures/fitlogChart.png b/docs/source/figures/fitlogChart.png new file mode 100644 index 00000000..57ae1683 Binary files /dev/null and b/docs/source/figures/fitlogChart.png differ diff --git a/docs/source/figures/fitlogTable.png b/docs/source/figures/fitlogTable.png new file mode 100644 index 00000000..37551634 Binary files /dev/null and b/docs/source/figures/fitlogTable.png differ diff --git a/docs/source/figures/workflow.png b/docs/source/figures/workflow.png new file mode 100644 index 00000000..d2f22df8 Binary files /dev/null and b/docs/source/figures/workflow.png differ diff --git a/docs/source/index.rst b/docs/source/index.rst index 9f410f41..03a192dc 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,63 +1,80 @@ -fastNLP documentation +fastNLP 中文文档 ===================== -A Modularized and Extensible Toolkit for Natural Language Processing. Currently still in incubation. +fastNLP 是一款轻量级的 NLP 处理套件。你既可以使用它快速地完成一个命名实体识别（NER）、中文分词或文本分类任务； +也可以使用他构建许多复杂的网络模型，进行科研。它具有如下的特性: -Introduction +- 统一的Tabular式数据容器，让数据预处理过程简洁明了。内置多种数据集的DataSet Loader，省去预处理代码。 +- 各种方便的NLP工具，例如预处理embedding加载; 中间数据cache等; +- 详尽的中文文档以供查阅； +- 提供诸多高级模块，例如Variational LSTM, Transformer, CRF等; +- 封装CNNText，Biaffine等模型可供直接使用; +- 便捷且具有扩展性的训练器; 提供多种内置callback函数，方便实验记录、异常捕获等。 + + +内置组件 ------------ -FastNLP is a modular Natural Language Processing system based on -PyTorch, built for fast development of NLP models. +大部分用于的 NLP 任务神经网络都可以看做由编码（encoder）、聚合（aggregator）、解码（decoder）三种模块组成。 -A deep learning NLP model is the composition of three types of modules: +.. image:: figures/text_classification.png + +fastNLP 在 :mod:`~fastNLP.modules` 模块中内置了三种模块的诸多组件，可以帮助用户快速搭建自己所需的网络。 +三种模块的功能和常见组件如下: +-----------------------+-----------------------+-----------------------+ | module type | functionality | example | +=======================+=======================+=======================+ -| encoder | encode the input into | embedding, RNN, CNN, | -| | some abstract | transformer | -| | representation | | +| encoder | 将输入编码为具有具 | embedding, RNN, CNN, | +| | 有表示能力的向量 | transformer | +-----------------------+-----------------------+-----------------------+ -| aggregator | aggregate and reduce | self-attention, | -| | information | max-pooling | +| aggregator | 从多个向量中聚合信息 | self-attention, | +| | | max-pooling | +-----------------------+-----------------------+-----------------------+ -| decoder | decode the | MLP, CRF | -| | representation into | | -| | the output | | +| decoder | 将具有某种表示意义的 | MLP, CRF | +| | 向量解码为需要的输出 | | +| | 形式 | | +-----------------------+-----------------------+-----------------------+ -For example: - -.. image:: figures/text_classification.png +内置模型 +---------------- +fastNLP 在 :mod:`~fastNLP.models` 模块中内置了如 :class:`~fastNLP.models.CNNText` 、 +:class:`~fastNLP.models.SeqLabeling` 等完整的模型，以供用户直接使用。 +.. todo:: + 这些模型的介绍如下表所示：（模型名称 + 介绍 + 任务上的结果） +用户手册 +---------------- -User's Guide ------------- .. toctree:: - :maxdepth: 2 + :maxdepth: 1 - user/installation - user/quickstart + 安装指南 + 快速入门 + 详细指南 + 科研指南 - -API Reference +API 文档 ------------- -If you are looking for information on a specific function, class or -method, this part of the documentation is for you. +除了用户手册之外，你还可以通过查阅 API 文档来找到你所需要的工具。 .. toctree:: + :titlesonly: :maxdepth: 2 - fastNLP API - + fastNLP +fitlog +------ +用户可以 `点此 `_ 查看fitlog的文档。 +fitlog 是由我们团队开发，用于帮助用户记录日志并管理代码的工具 -Indices and tables +索引与搜索 ================== * :ref:`genindex` diff --git a/docs/source/modules.rst b/docs/source/modules.rst new file mode 100644 index 00000000..9ca3c7f3 --- /dev/null +++ b/docs/source/modules.rst @@ -0,0 +1,8 @@ +fastNLP +======= + +.. toctree:: + :titlesonly: + :maxdepth: 4 + + fastNLP diff --git a/docs/source/tutorials/fastnlp_10tmin_tutorial.rst b/docs/source/tutorials/fastnlp_10tmin_tutorial.rst deleted file mode 100644 index 30293796..00000000 --- a/docs/source/tutorials/fastnlp_10tmin_tutorial.rst +++ /dev/null @@ -1,375 +0,0 @@ - -fastNLP上手教程 -=============== - -fastNLP提供方便的数据预处理，训练和测试模型的功能 - -DataSet & Instance ------------------- - -fastNLP用DataSet和Instance保存和处理数据。每个DataSet表示一个数据集，每个Instance表示一个数据样本。一个DataSet存有多个Instance，每个Instance可以自定义存哪些内容。 - -有一些read\_\*方法，可以轻松从文件读取数据，存成DataSet。 - -.. code:: ipython3 - - from fastNLP import DataSet - from fastNLP import Instance - - # 从csv读取数据到DataSet - win_path = "C:\\Users\zyfeng\Desktop\FudanNLP\\fastNLP\\test\\data_for_tests\\tutorial_sample_dataset.csv" - dataset = DataSet.read_csv(win_path, headers=('raw_sentence', 'label'), sep='\t') - print(dataset[0]) - - -.. parsed-literal:: - - {'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story ., - 'label': 1} - - -.. code:: ipython3 - - # DataSet.append(Instance)加入新数据 - - dataset.append(Instance(raw_sentence='fake data', label='0')) - dataset[-1] - - - - -.. parsed-literal:: - - {'raw_sentence': fake data, - 'label': 0} - - - -.. code:: ipython3 - - # DataSet.apply(func, new_field_name)对数据预处理 - - # 将所有数字转为小写 - dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence') - # label转int - dataset.apply(lambda x: int(x['label']), new_field_name='label_seq', is_target=True) - # 使用空格分割句子 - dataset.drop(lambda x: len(x['raw_sentence'].split()) == 0) - def split_sent(ins): - return ins['raw_sentence'].split() - dataset.apply(split_sent, new_field_name='words', is_input=True) - -.. code:: ipython3 - - # DataSet.drop(func)筛除数据 - # 删除低于某个长度的词语 - dataset.drop(lambda x: len(x['words']) <= 3) - -.. code:: ipython3 - - # 分出测试集、训练集 - - test_data, train_data = dataset.split(0.3) - print("Train size: ", len(test_data)) - print("Test size: ", len(train_data)) - - -.. parsed-literal:: - - Train size: 54 - Test size: - -Vocabulary ----------- - -fastNLP中的Vocabulary轻松构建词表，将词转成数字 - -.. code:: ipython3 - - from fastNLP import Vocabulary - - # 构建词表, Vocabulary.add(word) - vocab = Vocabulary(min_freq=2) - train_data.apply(lambda x: [vocab.add(word) for word in x['words']]) - vocab.build_vocab() - - # index句子, Vocabulary.to_index(word) - train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='word_seq', is_input=True) - test_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='word_seq', is_input=True) - - - print(test_data[0]) - - -.. parsed-literal:: - - {'raw_sentence': the plot is romantic comedy boilerplate from start to finish ., - 'label': 2, - 'label_seq': 2, - 'words': ['the', 'plot', 'is', 'romantic', 'comedy', 'boilerplate', 'from', 'start', 'to', 'finish', '.'], - 'word_seq': [2, 13, 9, 24, 25, 26, 15, 27, 11, 28, 3]} - - -.. code:: ipython3 - - # 假设你们需要做强化学习或者gan之类的项目，也许你们可以使用这里的dataset - from fastNLP.core.batch import Batch - from fastNLP.core.sampler import RandomSampler - - batch_iterator = Batch(dataset=train_data, batch_size=2, sampler=RandomSampler()) - for batch_x, batch_y in batch_iterator: - print("batch_x has: ", batch_x) - print("batch_y has: ", batch_y) - break - - -.. parsed-literal:: - - batch_x has: {'words': array([list(['this', 'kind', 'of', 'hands-on', 'storytelling', 'is', 'ultimately', 'what', 'makes', 'shanghai', 'ghetto', 'move', 'beyond', 'a', 'good', ',', 'dry', ',', 'reliable', 'textbook', 'and', 'what', 'allows', 'it', 'to', 'rank', 'with', 'its', 'worthy', 'predecessors', '.']), - list(['the', 'entire', 'movie', 'is', 'filled', 'with', 'deja', 'vu', 'moments', '.'])], - dtype=object), 'word_seq': tensor([[ 19, 184, 6, 1, 481, 9, 206, 50, 91, 1210, 1609, 1330, - 495, 5, 63, 4, 1269, 4, 1, 1184, 7, 50, 1050, 10, - 8, 1611, 16, 21, 1039, 1, 2], - [ 3, 711, 22, 9, 1282, 16, 2482, 2483, 200, 2, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0]])} - batch_y has: {'label_seq': tensor([3, 2])} - - -Model ------ - -.. code:: ipython3 - - # 定义一个简单的Pytorch模型 - - from fastNLP.models import CNNText - model = CNNText(embed_num=len(vocab), embed_dim=50, num_classes=5, padding=2, dropout=0.1) - model - - - - -.. parsed-literal:: - - CNNText( - (embed): Embedding( - (embed): Embedding(77, 50, padding_idx=0) - (dropout): Dropout(p=0.0) - ) - (conv_pool): ConvMaxpool( - (convs): ModuleList( - (0): Conv1d(50, 3, kernel_size=(3,), stride=(1,), padding=(2,)) - (1): Conv1d(50, 4, kernel_size=(4,), stride=(1,), padding=(2,)) - (2): Conv1d(50, 5, kernel_size=(5,), stride=(1,), padding=(2,)) - ) - ) - (dropout): Dropout(p=0.1) - (fc): Linear( - (linear): Linear(in_features=12, out_features=5, bias=True) - ) - ) - - - -Trainer & Tester ----------------- - -使用fastNLP的Trainer训练模型 - -.. code:: ipython3 - - from fastNLP import Trainer - from copy import deepcopy - from fastNLP import CrossEntropyLoss - from fastNLP import AccuracyMetric - -.. code:: ipython3 - - # 进行overfitting测试 - copy_model = deepcopy(model) - overfit_trainer = Trainer(model=copy_model, - train_data=test_data, - dev_data=test_data, - loss=CrossEntropyLoss(pred="output", target="label_seq"), - metrics=AccuracyMetric(), - n_epochs=10, - save_path=None) - overfit_trainer.train() - - -.. parsed-literal:: - - training epochs started 2018-12-07 14:07:20 - - - - -.. parsed-literal:: - - HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=20), HTML(value='')), layout=Layout(display='… - - - -.. parsed-literal:: - - Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.037037 - Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.296296 - Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.333333 - Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.555556 - Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.611111 - Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.481481 - Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.62963 - Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.685185 - Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.722222 - Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.777778 - - -.. code:: ipython3 - - # 实例化Trainer，传入模型和数据，进行训练 - trainer = Trainer(model=model, - train_data=train_data, - dev_data=test_data, - loss=CrossEntropyLoss(pred="output", target="label_seq"), - metrics=AccuracyMetric(), - n_epochs=5) - trainer.train() - print('Train finished!') - - -.. parsed-literal:: - - training epochs started 2018-12-07 14:08:10 - - - - -.. parsed-literal:: - - HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=5), HTML(value='')), layout=Layout(display='i… - - - -.. parsed-literal:: - - Epoch 1/5. Step:1/5. AccuracyMetric: acc=0.037037 - Epoch 2/5. Step:2/5. AccuracyMetric: acc=0.037037 - Epoch 3/5. Step:3/5. AccuracyMetric: acc=0.037037 - Epoch 4/5. Step:4/5. AccuracyMetric: acc=0.185185 - Epoch 5/5. Step:5/5. AccuracyMetric: acc=0.240741 - Train finished! - - -.. code:: ipython3 - - from fastNLP import Tester - - tester = Tester(data=test_data, model=model, metrics=AccuracyMetric()) - acc = tester.test() - - -.. parsed-literal:: - - [tester] - AccuracyMetric: acc=0.240741 - - -In summary ----------- - -fastNLP Trainer的伪代码逻辑 ---------------------------- - -1. 准备DataSet，假设DataSet中共有如下的fields -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - ['raw_sentence', 'word_seq1', 'word_seq2', 'raw_label','label'] - 通过 - DataSet.set_input('word_seq1', word_seq2', flag=True)将'word_seq1', 'word_seq2'设置为input - 通过 - DataSet.set_target('label', flag=True)将'label'设置为target - -2. 初始化模型 -~~~~~~~~~~~~~ - -:: - - class Model(nn.Module): - def __init__(self): - xxx - def forward(self, word_seq1, word_seq2): - # (1) 这里使用的形参名必须和DataSet中的input field的名称对应。因为我们是通过形参名, 进行赋值的 - # (2) input field的数量可以多于这里的形参数量。但是不能少于。 - xxxx - # 输出必须是一个dict - -3. Trainer的训练过程 -~~~~~~~~~~~~~~~~~~~~ - -:: - - (1) 从DataSet中按照batch_size取出一个batch，调用Model.forward - (2) 将 Model.forward的结果 与 标记为target的field 传入Losser当中。 - 由于每个人写的Model.forward的output的dict可能key并不一样，比如有人是{'pred':xxx}, {'output': xxx}; - 另外每个人将target可能也会设置为不同的名称, 比如有人是label, 有人设置为target； - 为了解决以上的问题，我们的loss提供映射机制 - 比如CrossEntropyLosser的需要的输入是(prediction, target)。但是forward的output是{'output': xxx}; 'label'是target - 那么初始化losser的时候写为CrossEntropyLosser(prediction='output', target='label')即可 - (3) 对于Metric是同理的 - Metric计算也是从 forward的结果中取值 与 设置target的field中取值。 也是可以通过映射找到对应的值 - -一些问题. ---------- - -1. DataSet中为什么需要设置input和target -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - 只有被设置为input或者target的数据才会在train的过程中被取出来 - (1.1) 我们只会在设置为input的field中寻找传递给Model.forward的参数。 - (1.2) 我们在传递值给losser或者metric的时候会使用来自: - (a)Model.forward的output - (b)被设置为target的field - - -2. 我们是通过forwad中的形参名将DataSet中的field赋值给对应的参数 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - (1.1) 构建模型过程中， - 例如: - DataSet中x，seq_lens是input，那么forward就应该是 - def forward(self, x, seq_lens): - pass - 我们是通过形参名称进行匹配的field的 - - -1. 加载数据到DataSet -~~~~~~~~~~~~~~~~~~~~ - -2. 使用apply操作对DataSet进行预处理 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - (2.1) 处理过程中将某些field设置为input，某些field设置为target - -3. 构建模型 -~~~~~~~~~~~ - -:: - - (3.1) 构建模型过程中，需要注意forward函数的形参名需要和DataSet中设置为input的field名称是一致的。 - 例如: - DataSet中x，seq_lens是input，那么forward就应该是 - def forward(self, x, seq_lens): - pass - 我们是通过形参名称进行匹配的field的 - (3.2) 模型的forward的output需要是dict类型的。 - 建议将输出设置为{"pred": xx}. - diff --git a/docs/source/tutorials/fastnlp_1_minute_tutorial.rst b/docs/source/tutorials/fastnlp_1_minute_tutorial.rst deleted file mode 100644 index b4471e00..00000000 --- a/docs/source/tutorials/fastnlp_1_minute_tutorial.rst +++ /dev/null @@ -1,111 +0,0 @@ - -FastNLP 1分钟上手教程 -===================== - -step 1 ------- - -读取数据集 - -.. code:: ipython3 - - from fastNLP import DataSet - # linux_path = "../test/data_for_tests/tutorial_sample_dataset.csv" - win_path = "C:\\Users\zyfeng\Desktop\FudanNLP\\fastNLP\\test\\data_for_tests\\tutorial_sample_dataset.csv" - ds = DataSet.read_csv(win_path, headers=('raw_sentence', 'label'), sep='\t') - -step 2 ------- - -数据预处理 1. 类型转换 2. 切分验证集 3. 构建词典 - -.. code:: ipython3 - - # 将所有数字转为小写 - ds.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence') - # label转int - ds.apply(lambda x: int(x['label']), new_field_name='label_seq', is_target=True) - - def split_sent(ins): - return ins['raw_sentence'].split() - ds.apply(split_sent, new_field_name='words', is_input=True) - - -.. code:: ipython3 - - # 分割训练集/验证集 - train_data, dev_data = ds.split(0.3) - print("Train size: ", len(train_data)) - print("Test size: ", len(dev_data)) - - -.. parsed-literal:: - - Train size: 54 - Test size: 23 - - -.. code:: ipython3 - - from fastNLP import Vocabulary - vocab = Vocabulary(min_freq=2) - train_data.apply(lambda x: [vocab.add(word) for word in x['words']]) - - # index句子, Vocabulary.to_index(word) - train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='word_seq', is_input=True) - dev_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='word_seq', is_input=True) - - -step 3 ------- - -定义模型 - -.. code:: ipython3 - - from fastNLP.models import CNNText - model = CNNText(embed_num=len(vocab), embed_dim=50, num_classes=5, padding=2, dropout=0.1) - - -step 4 ------- - -开始训练 - -.. code:: ipython3 - - from fastNLP import Trainer, CrossEntropyLoss, AccuracyMetric - trainer = Trainer(model=model, - train_data=train_data, - dev_data=dev_data, - loss=CrossEntropyLoss(), - metrics=AccuracyMetric() - ) - trainer.train() - print('Train finished!') - - - -.. parsed-literal:: - - training epochs started 2018-12-07 14:03:41 - - - - -.. parsed-literal:: - - HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=6), HTML(value='')), layout=Layout(display='i… - - - -.. parsed-literal:: - - Epoch 1/3. Step:2/6. AccuracyMetric: acc=0.26087 - Epoch 2/3. Step:4/6. AccuracyMetric: acc=0.347826 - Epoch 3/3. Step:6/6. AccuracyMetric: acc=0.608696 - Train finished! - - -本教程结束。更多操作请参考进阶教程。 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/user/installation.rst b/docs/source/user/installation.rst index 7dc39b3b..c218b3e1 100644 --- a/docs/source/user/installation.rst +++ b/docs/source/user/installation.rst @@ -1,16 +1,20 @@ -============ -Installation -============ +=============== +安装指南 +=============== .. contents:: :local: +fastNLP 依赖如下包:: -Run the following commands to install fastNLP package: + torch>=0.4.0 + numpy + tqdm + nltk -.. code:: shell - - pip install fastNLP - +其中torch的安装可能与操作系统及 CUDA 的版本相关，请参见 `PyTorch 官网 `_ 。 +在依赖包安装完成的情况，您可以在命令行执行如下指令完成安装 +.. code:: shell + >>> pip install fastNLP diff --git a/docs/source/user/quickstart.rst b/docs/source/user/quickstart.rst index baa49eef..43056a26 100644 --- a/docs/source/user/quickstart.rst +++ b/docs/source/user/quickstart.rst @@ -1,9 +1,124 @@ -Quickstart -========== +=============== +快速入门 +=============== -.. toctree:: - :maxdepth: 1 +这是一个简单的分类任务 (数据来源 `kaggle `_ )。 +给出一段文字，预测它的标签是0~4中的哪一个。 - ../tutorials/fastnlp_1_minute_tutorial - ../tutorials/fastnlp_10tmin_tutorial +我们可以使用 fastNLP 中 io 模块中的 :class:`~fastNLP.io.CSVLoader` 类，轻松地从 csv 文件读取我们的数据。 +.. code-block:: python + + from fastNLP.io import CSVLoader + + loader = CSVLoader(headers=('raw_sentence', 'label'), sep='\t') + dataset = loader.load("./sample_data/tutorial_sample_dataset.csv") + +此时的 `dataset[0]` 的值如下,可以看到，数据集中的每个数据包含 ``raw_sentence`` 和 ``label`` 两个字段，他们的类型都是 ``str``:: + + {'raw_sentence': A series of escapades demonstrating the adage that what is good for the + goose is also good for the gander , some of which occasionally amuses but none of which + amounts to much of a story . type=str, + 'label': 1 type=str} + + +我们使用 :class:`~fastNLP.DataSet` 类的 :meth:`~fastNLP.DataSet.apply` 方法将 ``raw_sentence`` 中字母变成小写，并将句子分词。 + +.. code-block:: python + + dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='sentence') + dataset.apply(lambda x: x['sentence'].split(), new_field_name='words', is_input=True) + +然后我们再用 :class:`~fastNLP.Vocabulary` 类来统计数据中出现的单词，并将单词序列转化为训练可用的数字序列。 + +.. code-block:: python + + from fastNLP import Vocabulary + vocab = Vocabulary(min_freq=2).from_dataset(dataset, field_name='words') + vocab.index_dataset(dataset, field_name='words',new_field_name='words') + +同时，我们也将原来 str 类型的标签转化为数字，并设置为训练中的标准答案 ``target`` + +.. code-block:: python + + dataset.apply(lambda x: int(x['label']), new_field_name='target', is_target=True) + +现在我们可以导入 fastNLP 内置的文本分类模型 :class:`~fastNLP.models.CNNText` ， + + +.. code-block:: python + + from fastNLP.models import CNNText + model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1) + +:class:`~fastNLP.models.CNNText` 的网络结构如下:: + + CNNText( + (embed): Embedding( + 177, 50 + (dropout): Dropout(p=0.0) + ) + (conv_pool): ConvMaxpool( + (convs): ModuleList( + (0): Conv1d(50, 3, kernel_size=(3,), stride=(1,), padding=(2,)) + (1): Conv1d(50, 4, kernel_size=(4,), stride=(1,), padding=(2,)) + (2): Conv1d(50, 5, kernel_size=(5,), stride=(1,), padding=(2,)) + ) + ) + (dropout): Dropout(p=0.1) + (fc): Linear(in_features=12, out_features=5, bias=True) + ) + +下面我们用 :class:`~fastNLP.DataSet` 类的 :meth:`~fastNLP.DataSet.split` 方法将数据集划分为 ``train_data`` 和 ``dev_data`` +两个部分，分别用于训练和验证 + +.. code-block:: python + + train_data, dev_data = dataset.split(0.2) + +最后我们用 fastNLP 的 :class:`~fastNLP.Trainer` 进行训练，训练的过程中需要传入模型 ``model`` ，训练数据集 ``train_data`` ， +验证数据集 ``dev_data`` ，损失函数 ``loss`` 和衡量标准 ``metrics`` 。 +其中损失函数使用的是 fastNLP 提供的 :class:`~fastNLP.CrossEntropyLoss` 损失函数; +衡量标准使用的是 fastNLP 提供的 :class:`~fastNLP.AccuracyMetric` 正确率指标。 + +.. code-block:: python + + from fastNLP import Trainer, CrossEntropyLoss, AccuracyMetric + + trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, + loss=CrossEntropyLoss(), metrics=AccuracyMetric()) + trainer.train() + +训练过程的输出如下:: + + input fields after batch(if batch size is 2): + words: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26]) + target fields after batch(if batch size is 2): + target: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + + training epochs started 2019-05-09-10-59-39 + Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.333333 + + Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.533333 + + Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.533333 + + Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.533333 + + Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.6 + + Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.8 + + Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.8 + + Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.733333 + + Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.733333 + + Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.733333 + + + In Epoch:6/Step:12, got best dev performance:AccuracyMetric: acc=0.8 + Reloaded the best model. + +这份教程只是简单地介绍了使用 fastNLP 工作的流程，具体的细节分析见 :doc:`/user/tutorial_one` \ No newline at end of file diff --git a/docs/source/user/tutorial_one.rst b/docs/source/user/tutorial_one.rst new file mode 100644 index 00000000..0c7be77d --- /dev/null +++ b/docs/source/user/tutorial_one.rst @@ -0,0 +1,371 @@ +=============== +详细指南 +=============== + +我们使用和 :doc:`/user/quickstart` 中一样的任务来进行详细的介绍。给出一段文字，预测它的标签是0~4中的哪一个 +(数据来源 `kaggle `_ )。 + +-------------- +数据处理 +-------------- + +数据读入 + 我们可以使用 fastNLP :mod:`fastNLP.io` 模块中的 :class:`~fastNLP.io.CSVLoader` 类，轻松地从 csv 文件读取我们的数据。 + 这里的 dataset 是 fastNLP 中 :class:`~fastNLP.DataSet` 类的对象 + + .. code-block:: python + + from fastNLP.io import CSVLoader + + loader = CSVLoader(headers=('raw_sentence', 'label'), sep='\t') + dataset = loader.load("./sample_data/tutorial_sample_dataset.csv") + + 除了读取数据外，fastNLP 还提供了读取其它文件类型的 Loader 类、读取 Embedding的 Loader 等。详见 :doc:`/fastNLP.io` 。 + +Instance 和 DataSet + fastNLP 中的 :class:`~fastNLP.DataSet` 类对象类似于二维表格，它的每一列是一个 :mod:`~fastNLP.core.field` + 每一行是一个 :mod:`~fastNLP.core.instance` 。我们可以手动向数据集中添加 :class:`~fastNLP.Instance` 类的对象 + + .. code-block:: python + + from fastNLP import Instance + + dataset.append(Instance(raw_sentence='fake data', label='0')) + + 此时的 ``dataset[-1]`` 的值如下,可以看到，数据集中的每个数据包含 ``raw_sentence`` 和 ``label`` 两个 + :mod:`~fastNLP.core.field` ，他们的类型都是 ``str`` :: + + {'raw_sentence': fake data type=str, 'label': 0 type=str} + +field 的修改 + 我们使用 :class:`~fastNLP.DataSet` 类的 :meth:`~fastNLP.DataSet.apply` 方法将 ``raw_sentence`` 中字母变成小写，并将句子分词。 + 同时也将 ``label`` :mod:`~fastNLP.core.field` 转化为整数并改名为 ``target`` + + .. code-block:: python + + dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='sentence') + dataset.apply_field(lambda x: x.split(), field_name='sentence', new_field_name='words') + dataset.apply(lambda x: int(x['label']), new_field_name='target') + + ``words`` 和 ``target`` 已经足够用于 :class:`~fastNLP.models.CNNText` 的训练了，但我们从其文档 + :class:`~fastNLP.models.CNNText` 中看到，在 :meth:`~fastNLP.models.CNNText.forward` 的时候，还可以传入可选参数 ``seq_len`` 。 + 所以，我们再使用 :meth:`~fastNLP.DataSet.apply_field` 方法增加一个名为 ``seq_len`` 的 :mod:`~fastNLP.core.field` 。 + + .. code-block:: python + + dataset.apply_field(lambda x: len(x), field_name='words', new_field_name='seq_len') + + 观察可知： :meth:`~fastNLP.DataSet.apply_field` 与 :meth:`~fastNLP.DataSet.apply` 类似， + 但所传入的 `lambda` 函数是针对一个 :class:`~fastNLP.Instance` 中的一个 :mod:`~fastNLP.core.field` 的； + 而 :meth:`~fastNLP.DataSet.apply` 所传入的 `lambda` 函数是针对整个 :class:`~fastNLP.Instance` 的。 + + .. note:: + `lambda` 函数即匿名函数，是 Python 的重要特性。 ``lambda x: len(x)`` 和下面的这个函数的作用相同:: + + def func_lambda(x): + return len(x) + + 你也可以编写复杂的函数做为 :meth:`~fastNLP.DataSet.apply_field` 与 :meth:`~fastNLP.DataSet.apply` 的参数 + +Vocabulary 的使用 + 我们再用 :class:`~fastNLP.Vocabulary` 类来统计数据中出现的单词，并使用 :meth:`~fastNLP.Vocabularyindex_dataset` + 将单词序列转化为训练可用的数字序列。 + + .. code-block:: python + + from fastNLP import Vocabulary + + vocab = Vocabulary(min_freq=2).from_dataset(dataset, field_name='words') + vocab.index_dataset(dataset, field_name='words',new_field_name='words') + +数据集分割 + 除了修改 :mod:`~fastNLP.core.field` 之外，我们还可以对 :class:`~fastNLP.DataSet` 进行分割，以供训练、开发和测试使用。 + 下面这段代码展示了 :meth:`~fastNLP.DataSet.split` 的使用方法（但实际应该放在后面两段改名和设置输入的代码之后） + + .. code-block:: python + + train_dev_data, test_data = dataset.split(0.1) + train_data, dev_data = train_dev_data.split(0.1) + len(train_data), len(dev_data), len(test_data) + +--------------------- +使用内置模型训练 +--------------------- + +内置模型的输入输出命名 + fastNLP内置了一些完整的神经网络模型，详见 :doc:`/fastNLP.models` , 我们使用其中的 :class:`~fastNLP.models.CNNText` 模型进行训练。 + 为了使用内置的 :class:`~fastNLP.models.CNNText`，我们必须修改 :class:`~fastNLP.DataSet` 中 :mod:`~fastNLP.core.field` 的名称。 + 在这个例子中模型输入 (forward方法的参数) 为 ``words`` 和 ``seq_len`` ; 预测输出为 ``pred`` ;标准答案为 ``target`` 。 + 具体的命名规范可以参考 :doc:`/fastNLP.core.const` 。 + + 如果不想查看文档，您也可以使用 :class:`~fastNLP.Const` 类进行命名。下面的代码展示了给 :class:`~fastNLP.DataSet` 中 + :mod:`~fastNLP.core.field` 改名的 :meth:`~fastNLP.DataSet.rename_field` 方法，以及 :class:`~fastNLP.Const` 类的使用方法。 + + .. code-block:: python + + from fastNLP import Const + + dataset.rename_field('words', Const.INPUT) + dataset.rename_field('seq_len', Const.INPUT_LEN) + dataset.rename_field('target', Const.TARGET) + + 在给 :class:`~fastNLP.DataSet` 中 :mod:`~fastNLP.core.field` 改名后，我们还需要设置训练所需的输入和目标，这里使用的是 + :meth:`~fastNLP.DataSet.set_input` 和 :meth:`~fastNLP.DataSet.set_target` 两个函数。 + + .. code-block:: python + + dataset.set_input(Const.INPUT, Const.INPUT_LEN) + dataset.set_target(Const.TARGET) + +快速训练 + 现在我们可以导入 fastNLP 内置的文本分类模型 :class:`~fastNLP.models.CNNText` ，并使用 :class:`~fastNLP.Trainer` 进行训练了 + （其中 ``loss`` 和 ``metrics`` 的定义，我们将在后续两段代码中给出）。 + + .. code-block:: python + + from fastNLP.models import CNNText + from fastNLP import Trainer + + model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1) + + trainer = Trainer(model=model_cnn, train_data=train_data, dev_data=dev_data, + loss=loss, metrics=metrics) + trainer.train() + + 训练过程的输出如下:: + + input fields after batch(if batch size is 2): + words: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26]) + target fields after batch(if batch size is 2): + target: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + + training epochs started 2019-05-09-10-59-39 + Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.333333 + + Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.533333 + + Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.533333 + + Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.533333 + + Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.6 + + Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.8 + + Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.8 + + Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.733333 + + Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.733333 + + Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.733333 + + + In Epoch:6/Step:12, got best dev performance:AccuracyMetric: acc=0.8 + Reloaded the best model. + +损失函数 + 训练模型需要提供一个损失函数, 下面提供了一个在分类问题中常用的交叉熵损失。注意它的 **初始化参数** 。 + ``pred`` 参数对应的是模型的 forward 方法返回的 dict 中的一个 key 的名字。 + ``target`` 参数对应的是 :class:`~fastNLP.DataSet` 中作为标签的 :mod:`~fastNLP.core.field` 的名字。 + 这里我们用 :class:`~fastNLP.Const` 来辅助命名，如果你自己编写模型中 forward 方法的返回值或 + 数据集中 :mod:`~fastNLP.core.field` 的名字与本例不同， 你可以把 ``pred`` 参数和 ``target`` 参数设定符合自己代码的值。 + + .. code-block:: python + + from fastNLP import CrossEntropyLoss + + # loss = CrossEntropyLoss() 在本例中与下面这行代码等价 + loss = CrossEntropyLoss(pred=Const.OUTPUT, target=Const.TARGET) + +评价指标 + 训练模型需要提供一个评价指标。这里使用准确率做为评价指标。参数的 `命名规则` 跟上面类似。 + ``pred`` 参数对应的是模型的 forward 方法返回的 dict 中的一个 key 的名字。 + ``target`` 参数对应的是 :class:`~fastNLP.DataSet` 中作为标签的 :mod:`~fastNLP.core.field` 的名字。 + + .. code-block:: python + + from fastNLP import AccuracyMetric + + # metrics=AccuracyMetric() 在本例中与下面这行代码等价 + metrics=AccuracyMetric(pred=Const.OUTPUT, target=Const.TARGET) + +快速测试 + 与 :class:`~fastNLP.Trainer` 对应，fastNLP 也提供了 :class:`~fastNLP.Tester` 用于快速测试，用法如下 + + .. code-block:: python + + from fastNLP import Tester + + tester = Tester(test_data, model_cnn, metrics=AccuracyMetric()) + tester.test() + +--------------------- +编写自己的模型 +--------------------- + +因为 fastNLP 是基于 `PyTorch `_ 开发的框架，所以我们可以基于 PyTorch 模型编写自己的神经网络模型。 +与标准的 PyTorch 模型不同，fastNLP 模型中 forward 方法返回的是一个字典，字典中至少需要包含 "pred" 这个字段。 +而 forward 方法的参数名称必须与 :class:`~fastNLP.DataSet` 中用 :meth:`~fastNLP.DataSet.set_input` 设定的名称一致。 +模型定义的代码如下: + +.. code-block:: python + + import torch + import torch.nn as nn + + class LSTMText(nn.Module): + def __init__(self, vocab_size, embedding_dim, output_dim, hidden_dim=64, num_layers=2, dropout=0.5): + super().__init__() + + self.embedding = nn.Embedding(vocab_size, embedding_dim) + self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=num_layers, bidirectional=True, dropout=dropout) + self.fc = nn.Linear(hidden_dim * 2, output_dim) + self.dropout = nn.Dropout(dropout) + + def forward(self, words): + # (input) words : (batch_size, seq_len) + words = words.permute(1,0) + # words : (seq_len, batch_size) + + embedded = self.dropout(self.embedding(words)) + # embedded : (seq_len, batch_size, embedding_dim) + output, (hidden, cell) = self.lstm(embedded) + # output: (seq_len, batch_size, hidden_dim * 2) + # hidden: (num_layers * 2, batch_size, hidden_dim) + # cell: (num_layers * 2, batch_size, hidden_dim) + + hidden = torch.cat((hidden[-2, :, :], hidden[-1, :, :]), dim=1) + hidden = self.dropout(hidden) + # hidden: (batch_size, hidden_dim * 2) + + pred = self.fc(hidden.squeeze(0)) + # result: (batch_size, output_dim) + return {"pred":pred} + +模型的使用方法与内置模型 :class:`~fastNLP.models.CNNText` 一致 + +.. code-block:: python + + model_lstm = LSTMText(len(vocab),50,5) + + trainer = Trainer(model=model_lstm, train_data=train_data, dev_data=dev_data, + loss=loss, metrics=metrics) + trainer.train() + + tester = Tester(test_data, model_lstm, metrics=AccuracyMetric()) + tester.test() + +.. todo:: + 使用 :doc:`/fastNLP.modules` 编写模型 + +-------------------------- +自己编写训练过程 +-------------------------- + +如果你想用类似 PyTorch 的使用方法，自己编写训练过程，你可以参考下面这段代码。其中使用了 fastNLP 提供的 :class:`~fastNLP.Batch` +来获得小批量训练的小批量数据，使用 :class:`~fastNLP.BucketSampler` 做为 :class:`~fastNLP.Batch` 的参数来选择采样的方式。 +这段代码中使用了 PyTorch 的 `torch.optim.Adam` 优化器 和 `torch.nn.CrossEntropyLoss` 损失函数，并自己计算了正确率 + +.. code-block:: python + + from fastNLP import BucketSampler + from fastNLP import Batch + import torch + import time + + model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1) + + def train(epoch, data): + optim = torch.optim.Adam(model.parameters(), lr=0.001) + lossfunc = torch.nn.CrossEntropyLoss() + batch_size = 32 + + train_sampler = BucketSampler(batch_size=batch_size, seq_len_field_name='seq_len') + train_batch = Batch(batch_size=batch_size, dataset=data, sampler=train_sampler) + + start_time = time.time() + for i in range(epoch): + loss_list = [] + for batch_x, batch_y in train_batch: + optim.zero_grad() + output = model(batch_x['words']) + loss = lossfunc(output['pred'], batch_y['target']) + loss.backward() + optim.step() + loss_list.append(loss.item()) + print('Epoch {:d} Avg Loss: {:.2f}'.format(i, sum(loss_list) / len(loss_list)),end=" ") + print('{:d}ms'.format(round((time.time()-start_time)*1000))) + loss_list.clear() + + train(10, train_data) + + tester = Tester(test_data, model, metrics=AccuracyMetric()) + tester.test() + +这段代码的输出如下:: + + Epoch 0 Avg Loss: 2.76 17ms + Epoch 1 Avg Loss: 2.55 29ms + Epoch 2 Avg Loss: 2.37 41ms + Epoch 3 Avg Loss: 2.30 53ms + Epoch 4 Avg Loss: 2.12 65ms + Epoch 5 Avg Loss: 2.16 76ms + Epoch 6 Avg Loss: 1.88 88ms + Epoch 7 Avg Loss: 1.84 99ms + Epoch 8 Avg Loss: 1.71 111ms + Epoch 9 Avg Loss: 1.62 122ms + [tester] + AccuracyMetric: acc=0.142857 + +---------------------------------- +使用 Callback 增强 Trainer +---------------------------------- + +如果你不想自己实现繁琐的训练过程，只希望在训练过程中实现一些自己的功能（比如：输出从训练开始到当前 batch 结束的总时间）， +你可以使用 fastNLP 提供的 :class:`~fastNLP.Callback` 类。下面的例子中，我们继承 :class:`~fastNLP.Callback` 类实现了这个功能。 + +.. code-block:: python + + from fastNLP import Callback + + start_time = time.time() + + class MyCallback(Callback): + def on_epoch_end(self): + print('Sum Time: {:d}ms\n\n'.format(round((time.time()-start_time)*1000))) + + + model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1) + trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, + loss=CrossEntropyLoss(), metrics=AccuracyMetric(), callbacks=[MyCallback()]) + trainer.train() + +训练输出如下:: + + input fields after batch(if batch size is 2): + words: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 16]) + seq_len: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + target fields after batch(if batch size is 2): + target: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + + training epochs started 2019-05-12-21-38-40 + Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.285714 + + Sum Time: 51ms + + + ………………………… + + + Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.857143 + + Sum Time: 212ms + + + + In Epoch:10/Step:20, got best dev performance:AccuracyMetric: acc=0.857143 + Reloaded the best model. + +这个例子只是介绍了 :class:`~fastNLP.Callback` 类的使用方法。实际应用（比如：负采样、Learning Rate Decay、Early Stop 等）中 +很多功能已经被 fastNLP 实现了。你可以直接 import 它们使用，详细请查看文档 :doc:`/fastNLP.core.callback` 。 \ No newline at end of file diff --git a/docs/source/user/with_fitlog.rst b/docs/source/user/with_fitlog.rst new file mode 100644 index 00000000..51445775 --- /dev/null +++ b/docs/source/user/with_fitlog.rst @@ -0,0 +1,122 @@ +================= +科研向导 +================= + +本文介绍结合使用 fastNLP 和 fitlog 进行科研的方法。 + +首先，我们需要安装 `fitlog `_ 。你需要确认你的电脑中没有其它名为 `fitlog` 的命令。 + +我们从命令行中进入到一个文件夹，现在我们要在文件夹中创建我们的 fastNLP 项目。你可以在命令行输入 `fitlog init test1` ， +然后你会看到如下提示:: + + Initialized empty Git repository in /Users/fdujyn/workspaces/test1/.git/ + Auto commit by fitlog + Initialized empty Git repository in /Users/fdujyn/workspaces/test1/.git/ + Fitlog project test1 is initialized. + +这表明你已经创建成功了项目文件夹，并且在项目文件夹中已经初始化了 Git。如果你不想初始化 Git， +可以参考文档 `命令行工具 `_ + +现在我们进入你创建的项目文件夹 test1 中，可以看到有一个名为 logs 的文件夹，后面我们将会在里面存放你的实验记录。 +同时也有一个名为 main.py 的文件，这是我们推荐你使用的训练入口文件。文件的内容如下:: + + import fitlog + + fitlog.commit(__file__) # auto commit your codes + fitlog.add_hyper_in_file (__file__) # record your hyperparameters + + """ + Your training code here, you may use these functions to log your result: + fitlog.add_hyper() + fitlog.add_loss() + fitlog.add_metric() + fitlog.add_best_metric() + ...... + """ + + fitlog.finish() # finish the logging + +我们推荐你保留除注释外的四行代码，它们有助于你的实验， +他们的具体用处参见文档 `用户 API `_ + +我们假定你要进行前两个教程中的实验，并已经把数据复制到了项目根目录下的 tutorial_sample_dataset.csv 文件中。 +现在我们编写如下的训练代码，使用 :class:`~fastNLP.core.callback.FitlogCallback` 进行实验记录保存:: + + import fitlog + from fastNLP import Vocabulary, Trainer, CrossEntropyLoss, AccuracyMetric + from fastNLP.io import CSVLoader + from fastNLP.models import CNNText + from fastNLP.core.callback import FitlogCallback + + fitlog.commit(__file__) # auto commit your codes + fitlog.add_hyper_in_file (__file__) # record your hyperparameters + + ############hyper + word_embed = 50 + dropout = 0.1 + ############hyper + + loader = CSVLoader(headers=('raw_sentence', 'label'), sep='\t') + dataset = loader.load("tutorial_sample_dataset.csv") + + dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='sentence') + dataset.apply(lambda x: x['sentence'].split(), new_field_name='words', is_input=True) + dataset.apply(lambda x: int(x['label']), new_field_name='target', is_target=True) + vocab = Vocabulary(min_freq=2).from_dataset(dataset, field_name='words') + vocab.index_dataset(dataset, field_name='words',new_field_name='words') + + model = CNNText((len(vocab),word_embed), num_classes=5, padding=2, dropout=dropout) + + train_dev_data, test_data = dataset.split(0.1) + train_data, dev_data = train_dev_data.split(0.1) + + trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, + loss=CrossEntropyLoss(), metrics=AccuracyMetric(), + callbacks=[FitlogCallback(test_data)]) + trainer.train() + + fitlog.finish() # finish the logging + +用命令行在项目目录下执行 `python main.py` 之后，输出结果如下:: + + Auto commit by fitlog + input fields after batch(if batch size is 2): + words: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 11]) + target fields after batch(if batch size is 2): + target: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + + training epochs started 2019-05-23-21-11-51 + Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.285714 + + Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.285714 + + Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.285714 + + Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.428571 + + Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.571429 + + Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.571429 + + Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.285714 + + Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.142857 + + Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.285714 + + Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.571429 + + + In Epoch:5/Step:10, got best dev performance:AccuracyMetric: acc=0.571429 + Reloaded the best model. + +现在，我们在项目目录下输入 `fitlog log logs` ，命令行会启动一个网页，默认 url 为 ``0.0.0.0:5000`` 。 +我们在浏览器中打开网页，可以看到如下的统计表格: + +.. image:: ../figures/fitlogTable.png + +如果我们点击action中的最后一个键钮，可以看到详细的 loss 图: + +.. image:: ../figures/fitlogChart.png + +更多的教程还在编写中，敬请期待~ \ No newline at end of file diff --git a/fastNLP/__init__.py b/fastNLP/__init__.py index 0f6da45f..c67e5919 100644 --- a/fastNLP/__init__.py +++ b/fastNLP/__init__.py @@ -1,3 +1,59 @@ +""" +fastNLP 由 :mod:`~fastNLP.core` 、 :mod:`~fastNLP.io` 、:mod:`~fastNLP.modules`、:mod:`~fastNLP.models` +等子模块组成，你可以点进去查看每个模块的文档。 + +- :mod:`~fastNLP.core` 是fastNLP 的核心模块，包括 DataSet、 Trainer、 Tester 等组件。详见文档 :doc:`/fastNLP.core` +- :mod:`~fastNLP.io` 是实现输入输出的模块，包括了数据集的读取，模型的存取等功能。详见文档 :doc:`/fastNLP.io` +- :mod:`~fastNLP.modules` 包含了用于搭建神经网络模型的诸多组件，可以帮助用户快速搭建自己所需的网络。详见文档 :doc:`/fastNLP.modules` +- :mod:`~fastNLP.models` 包含了一些使用 fastNLP 实现的完整网络模型，包括CNNText、SeqLabeling等常见模型。详见文档 :doc:`/fastNLP.models` + +fastNLP 中最常用的组件可以直接从 fastNLP 包中 import ，他们的文档如下： +""" +__all__ = [ + "Instance", + "FieldArray", + "Batch", + "Vocabulary", + "DataSet", + "Const", + + "Trainer", + "Tester", + + "Callback", + "GradientClipCallback", + "EarlyStopCallback", + "TensorboardCallback", + "LRScheduler", + "ControlC", + + "Padder", + "AutoPadder", + "EngChar2DPadder", + + "AccuracyMetric", + "SpanFPreRecMetric", + "SQuADMetric", + + "Optimizer", + "SGD", + "Adam", + + "Sampler", + "SequentialSampler", + "BucketSampler", + "RandomSampler", + + "LossFunc", + "CrossEntropyLoss", + "L1Loss", "BCELoss", + "NLLLoss", + "LossInForward", + + "cache_results" +] +__version__ = '0.4.0' + from .core import * from . import models from . import modules diff --git a/fastNLP/api/README.md b/fastNLP/api/README.md deleted file mode 100644 index 3604bd07..00000000 --- a/fastNLP/api/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# fastNLP 高级接口 - -### 环境与配置 -1. 系统环境：linux/ubuntu(推荐) -2. 编程语言：Python>=3.6 -3. Python包依赖 - - **torch==1.0** - - numpy>=1.14.2 - -### 中文分词 -```python -text = ['编者按：7月12日，英国航空航天系统公司公布了该公司研制的第一款高科技隐形无人机雷电之神。', - '这款飞行从外型上来看酷似电影中的太空飞行器，据英国方面介绍，可以实现洲际远程打击。', - '那么这款无人机到底有多厉害？'] -from fastNLP.api import CWS -cws = CWS(device='cpu') -print(cws.predict(text)) -# ['编者 按 : 7月 12日 , 英国 航空 航天 系统 公司 公布 了 该 公司 研制 的 第一 款 高 科技 隐形 无人 机雷电 之 神 。', '这 款 飞行 从 外型 上 来 看 酷似 电影 中 的 太空 飞行器 , 据 英国 方面 介绍 , 可以 实现 洲际 远程 打击 。', '那么 这 款 无人 机 到底 有 多 厉害 ?'] -``` - -### 中文分词+词性标注 -```python -text = ['编者按：7月12日，英国航空航天系统公司公布了该公司研制的第一款高科技隐形无人机雷电之神。', - '这款飞行从外型上来看酷似电影中的太空飞行器，据英国方面介绍，可以实现洲际远程打击。', - '那么这款无人机到底有多厉害？'] -from fastNLP.api import POS -pos = POS(device='cpu') -print(pos.predict(text)) -# [['编者/NN', '按/P', '：/PU', '7月/NT', '12日/NR', '，/PU', '英国/NR', '航空/NN', '航天/NN', '系统/NN', '公司/NN', '公布/VV', '了/AS', '该/DT', '公司/NN', '研制/VV', '的/DEC', '第一/OD', '款高/NN', '科技/NN', '隐形/NN', '无/VE', '人机/NN', '雷电/NN', '之/DEG', '神/NN', '。/PU'], ['这/DT', '款/NN', '飞行/VV', '从/P', '外型/NN', '上/LC', '来/MSP', '看/VV', '酷似/VV', '电影/NN', '中/LC', '的/DEG', '太空/NN', '飞行器/NN', '，/PU', '据/P', '英国/NR', '方面/NN', '介绍/VV', '，/PU', '可以/VV', '实现/VV', '洲际/NN', '远程/NN', '打击/NN', '。/PU'], ['那么/AD', '这/DT', '款/NN', '无/VE', '人机/NN', '到底/AD', '有/VE', '多/CD', '厉害/NN', '？/PU']] -``` - -### 中文分词+词性标注+句法分析 -```python -text = ['编者按：7月12日，英国航空航天系统公司公布了该公司研制的第一款高科技隐形无人机雷电之神。', - '这款飞行从外型上来看酷似电影中的太空飞行器，据英国方面介绍，可以实现洲际远程打击。', - '那么这款无人机到底有多厉害？'] -from fastNLP.api import Parser -parser = Parser(device='cpu') -print(parser.predict(text)) -# [['12/nsubj', '12/prep', '2/punct', '5/nn', '2/pobj', '12/punct', '11/nn', '11/nn', '11/nn', '11/nn', '2/pobj', '0/root', '12/asp', '15/det', '16/nsubj', '21/rcmod', '16/cpm', '21/nummod', '21/nn', '21/nn', '22/top', '12/ccomp', '24/nn', '26/assmod', '24/assm', '22/dobj', '12/punct'], ['2/det', '8/xsubj', '8/mmod', '8/prep', '6/lobj', '4/plmod', '8/prtmod', '0/root', '8/ccomp', '11/lobj', '14/assmod', '11/assm', '14/nn', '9/dobj', '8/punct', '22/prep', '18/nn', '19/nsubj', '16/pccomp', '22/punct', '22/mmod', '8/dep', '25/nn', '25/nn', '22/dobj', '8/punct'], ['4/advmod', '3/det', '4/nsubj', '0/root', '4/dobj', '7/advmod', '4/conj', '9/nummod', '7/dobj', '4/punct']] -``` - -完整样例见`examples.py` \ No newline at end of file diff --git a/fastNLP/api/__init__.py b/fastNLP/api/__init__.py deleted file mode 100644 index a21a4c42..00000000 --- a/fastNLP/api/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from .api import CWS, POS, Parser diff --git a/fastNLP/api/examples.py b/fastNLP/api/examples.py deleted file mode 100644 index 10cc6edc..00000000 --- a/fastNLP/api/examples.py +++ /dev/null @@ -1,29 +0,0 @@ -""" -api/example.py contains all API examples provided by fastNLP. -It is used as a tutorial for API or a test script since it is difficult to test APIs in travis. - -""" -from fastNLP.api import CWS, POS, Parser - -text = ['编者按：7月12日，英国航空航天系统公司公布了该公司研制的第一款高科技隐形无人机雷电之神。', - '这款飞行从外型上来看酷似电影中的太空飞行器，据英国方面介绍，可以实现洲际远程打击。', - '那么这款无人机到底有多厉害？'] - - -def chinese_word_segmentation(): - cws = CWS(device='cpu') - print(cws.predict(text)) - - -def pos_tagging(): - pos = POS(device='cpu') - print(pos.predict(text)) - - -def syntactic_parsing(): - parser = Parser(device='cpu') - print(parser.predict(text)) - - -if __name__ == "__main__": - syntactic_parsing() diff --git a/fastNLP/core/__init__.py b/fastNLP/core/__init__.py index 038ca12f..d6ab8983 100644 --- a/fastNLP/core/__init__.py +++ b/fastNLP/core/__init__.py @@ -1,13 +1,30 @@ +""" +core 模块里实现了 fastNLP 的核心框架，常用的功能都可以从 fastNLP 包中直接 import。当然你也同样可以从 core 模块的子模块中 import， +例如 Batch 组件有两种 import 的方式:: + + # 直接从 fastNLP 中 import + from fastNLP import Batch + + # 从 core 模块的子模块 batch 中 import + from fastNLP.core.batch import Batch + +对于常用的功能，你只需要在 :doc:`fastNLP` 中查看即可。如果想了解各个子模块的具体作用，您可以在下面找到每个子模块的具体文档。 + +.. todo:: + 介绍core 的子模块的分工，好像必要性不大 + +""" from .batch import Batch -# from .dataset import DataSet -from .fieldarray import FieldArray +from .callback import Callback, GradientClipCallback, EarlyStopCallback, TensorboardCallback, LRScheduler, ControlC +from .const import Const +from .dataset import DataSet +from .field import FieldArray, Padder, AutoPadder, EngChar2DPadder from .instance import Instance from .losses import LossFunc, CrossEntropyLoss, L1Loss, BCELoss, NLLLoss, LossInForward -from .metrics import AccuracyMetric +from .metrics import AccuracyMetric, SpanFPreRecMetric, SQuADMetric from .optimizer import Optimizer, SGD, Adam -from .sampler import SequentialSampler, BucketSampler, RandomSampler, BaseSampler +from .sampler import SequentialSampler, BucketSampler, RandomSampler, Sampler from .tester import Tester from .trainer import Trainer +from .utils import cache_results, seq_len_to_mask from .vocabulary import Vocabulary -from ..io.dataset_loader import DataSet - diff --git a/fastNLP/core/batch.py b/fastNLP/core/batch.py index 9ba8dca8..109d4fe9 100644 --- a/fastNLP/core/batch.py +++ b/fastNLP/core/batch.py @@ -1,74 +1,185 @@ +""" +batch 模块实现了 fastNLP 所需的 Batch 类。 + +""" +__all__ = [ + "Batch" +] + +import atexit +from queue import Empty, Full + import numpy as np import torch +import torch.multiprocessing as mp -from fastNLP.core.sampler import RandomSampler +from .sampler import RandomSampler +_python_is_exit = False -class Batch(object): - """Batch is an iterable object which iterates over mini-batches. - Example:: +def _set_python_is_exit(): + global _python_is_exit + _python_is_exit = True - for batch_x, batch_y in Batch(data_set, batch_size=16, sampler=SequentialSampler()): - # ... - :param DataSet dataset: a DataSet object - :param int batch_size: the size of the batch - :param Sampler sampler: a Sampler object - :param bool as_numpy: If True, return Numpy array. Otherwise, return torch tensors. +atexit.register(_set_python_is_exit) - """ - def __init__(self, dataset, batch_size, sampler=RandomSampler(), as_numpy=False): +class Batch(object): + """ + 别名：:class:`fastNLP.Batch` :class:`fastNLP.core.batch.Batch` + + Batch 用于从 `DataSet` 中按一定的顺序, 依次按 ``batch_size`` 的大小将数据取出， + 组成 `x` 和 `y`:: + + batch = Batch(data_set, batch_size=16, sampler=SequentialSampler()) + num_batch = len(batch) + for batch_x, batch_y in batch: + # do stuff ... + + :param dataset: :class:`~fastNLP.DataSet` 对象, 数据集 + :param int batch_size: 取出的batch大小 + :param sampler: 规定使用的 :class:`~fastNLP.Sampler` 方式. 若为 ``None`` , 使用 :class:`~fastNLP.RandomSampler`. + + Default: ``None`` + :param bool as_numpy: 若为 ``True`` , 输出batch为 numpy.array. 否则为 :class:`torch.Tensor`. + + Default: ``False`` + :param bool prefetch: 若为 ``True`` 使用多进程预先取出下一batch. + + Default: ``False`` + """ + + def __init__(self, dataset, batch_size, sampler=None, as_numpy=False, prefetch=False): self.dataset = dataset self.batch_size = batch_size + if sampler is None: + sampler = RandomSampler() self.sampler = sampler self.as_numpy = as_numpy self.idx_list = None self.curidx = 0 self.num_batches = len(dataset) // batch_size + int(len(dataset) % batch_size != 0) self.cur_batch_indices = None - - def __iter__(self): - self.idx_list = self.sampler(self.dataset) - self.curidx = 0 - self.lengths = self.dataset.get_length() - return self - - def __next__(self): + self.prefetch = prefetch + self.lengths = 0 + + def fetch_one(self): if self.curidx >= len(self.idx_list): - raise StopIteration + return None else: endidx = min(self.curidx + self.batch_size, len(self.idx_list)) batch_x, batch_y = {}, {} - + indices = self.idx_list[self.curidx:endidx] self.cur_batch_indices = indices - + for field_name, field in self.dataset.get_all_fields().items(): if field.is_target or field.is_input: batch = field.get(indices) - if not self.as_numpy: - batch = to_tensor(batch, field.dtype) + if not self.as_numpy and field.padder is not None: + batch = _to_tensor(batch, field.dtype) if field.is_target: batch_y[field_name] = batch if field.is_input: batch_x[field_name] = batch - + self.curidx = endidx - return batch_x, batch_y - + + def __iter__(self): + """ + Iterate on dataset, fetch batch data. Fetch process don't block the iterate process + :return: + """ + if self.prefetch: + return self._run_batch_iter(self) + + def batch_iter(): + self.init_iter() + while 1: + res = self.fetch_one() + if res is None: + break + yield res + + return batch_iter() + + def init_iter(self): + self.idx_list = self.sampler(self.dataset) + self.curidx = 0 + self.lengths = self.dataset.get_length() + def __len__(self): return self.num_batches - + def get_batch_indices(self): - return self.cur_batch_indices - + """ + 取得当前batch在DataSet中所在的index下标序列 -def to_tensor(batch, dtype): - if dtype in (int, np.int8, np.int16, np.int32, np.int64): - batch = torch.LongTensor(batch) - if dtype in (float, np.float32, np.float64): - batch = torch.FloatTensor(batch) + :return list(int) indexes: 下标序列 + """ + return self.cur_batch_indices + + @staticmethod + def _run_fetch(batch, q): + try: + global _python_is_exit + batch.init_iter() + # print('start fetch') + while 1: + res = batch.fetch_one() + # print('fetch one') + while 1: + try: + q.put(res, timeout=3) + break + except Full: + if _python_is_exit: + return + if res is None: + # print('fetch done, waiting processing') + break + # print('fetch exit') + except Exception as e: + q.put(e) + finally: + q.join() + + @staticmethod + def _run_batch_iter(batch): + q = mp.JoinableQueue(maxsize=10) + fetch_p = mp.Process(target=Batch._run_fetch, args=(batch, q)) + fetch_p.daemon = True + fetch_p.start() + # print('fork fetch process') + while 1: + try: + res = q.get(timeout=1) + q.task_done() + # print('get fetched') + if res is None: + break + elif isinstance(res, Exception): + raise res + yield res + except Empty as e: + if fetch_p.is_alive(): + continue + else: + break + fetch_p.terminate() + fetch_p.join() + # print('iter done') + + +def _to_tensor(batch, dtype): + try: + if dtype in (int, np.int8, np.int16, np.int32, np.int64): + batch = torch.LongTensor(batch) + if dtype in (float, np.float32, np.float64): + batch = torch.FloatTensor(batch) + except: + pass return batch diff --git a/fastNLP/core/callback.py b/fastNLP/core/callback.py index ce9627ea..e617cf2a 100644 --- a/fastNLP/core/callback.py +++ b/fastNLP/core/callback.py @@ -1,118 +1,301 @@ -class Callback(object): - """An Interface for all callbacks. +r""" +callback模块实现了 fastNLP 中的许多 callback 类，用于增强 :class:`~fastNLP.Trainer` 类。 + +虽然Trainer本身已经集成了一些功能，但仍然不足以囊括训练过程中可能需要到的功能， +比如负采样，learning rate decay, Early Stop等。 +为了解决这个问题fastNLP引入了callback的机制，Callback 是一种在Trainer训练过程中特定阶段会运行的函数集合。 +关于Trainer的详细文档，请参见 :doc:`trainer 模块` + +我们将 :meth:`~fastNLP.Train.train` 这个函数内部分为以下的阶段，在对应阶段会触发相应的调用:: + + callback.on_train_begin() # 开始进行训练 + for i in range(1, n_epochs+1): + callback.on_epoch_begin() # 开始新的epoch + for batch_x, batch_y in Batch: + callback.on_batch_begin(batch_x, batch_y, indices) # batch_x是设置为input的field，batch_y是设置为target的field + 获取模型输出 + callback.on_loss_begin() + 计算loss + callback.on_backward_begin() # 可以进行一些检查，比如loss是否为None + 反向梯度回传 + callback.on_backward_end() # 进行梯度截断等 + 进行参数更新 + callback.on_step_end() + callback.on_batch_end() + # 根据设置进行evaluation，比如这是本epoch最后一个batch或者达到一定step + if do evaluation: + callback.on_valid_begin() + 进行dev data上的验证 + callback.on_valid_end() # 可以进行在其它数据集上进行验证 + callback.on_epoch_end() # epoch结束调用 + callback.on_train_end() # 训练结束 + callback.on_exception() # 这是一个特殊的步骤，在训练过程中遭遇exception会跳转到这里。 + +如下面的例子所示，我们可以使用内置的 callback 类，或者继承 :class:`~fastNLP.core.callback.Callback` +定义自己的 callback 类:: + + from fastNLP import Callback, EarlyStopCallback, Trainer, CrossEntropyLoss, AccuracyMetric + from fastNLP.models import CNNText + + start_time = time.time() + + class MyCallback(Callback): + def on_epoch_end(self): + print('{:d}ms\n\n'.format(round((time.time()-start_time)*1000))) + + model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1) + trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, loss=CrossEntropyLoss(), + metrics=AccuracyMetric(), callbacks=[MyCallback(),EarlyStopCallback(10)]) + trainer.train() + +""" +__all__ = [ + "Callback", + "GradientClipCallback", + "EarlyStopCallback", + "TensorboardCallback", + "FitlogCallback", + "LRScheduler", + "ControlC", + + "CallbackException", + "EarlyStopError" +] + +import os + +import torch +from copy import deepcopy + +try: + from tensorboardX import SummaryWriter + + tensorboardX_flag = True +except: + tensorboardX_flag = False + +from ..io.model_io import ModelSaver, ModelLoader +from .dataset import DataSet +from .tester import Tester + +try: + import fitlog +except: + pass - Any customized callback should implement at least one of the following methods. +class Callback(object): """ + 别名：:class:`fastNLP.Callback` :class:`fastNLP.core.callback.Callback` + Callback是fastNLP中被设计用于增强 :class:`~fastNLP.Trainer` 的类。 + 如果Callback被传递给了 Trainer , 则 Trainer 会在对应的阶段调用Callback的函数， + 具体调用时机可以通过 :doc:`trainer 模块` 查看。 + 这是Callback的基类，所有的callback必须继承自这个类 + + """ + def __init__(self): super(Callback, self).__init__() + self._trainer = None # 在Trainer内部被重新赋值 + + @property + def trainer(self): + """ + 该属性可以通过self.trainer获取到，一般情况下不需要使用这个属性。 + """ + return self._trainer + + @property + def step(self): + """当前运行到的step, 范围为[1, self.n_steps+1)""" + return self._trainer.step + + @property + def n_steps(self): + """Trainer一共会运行多少步""" + return self._trainer.n_steps + + @property + def batch_size(self): + """train和evaluate时的batch_size为多大""" + return self._trainer.batch_size + + @property + def epoch(self): + """当前运行的epoch数，范围是[1, self.n_epochs+1)""" + return self._trainer.epoch + + @property + def n_epochs(self): + """一共会运行多少个epoch""" + return self._trainer.n_epochs + + @property + def optimizer(self): + """初始化Trainer时传递的Optimizer""" + return self._trainer.optimizer + + @property + def model(self): + """正在被Trainer训练的模型""" + return self._trainer.model + + @property + def pbar(self): + """如果在Callback中需要打印内容，请使用self.pbar.write(str)。否则可能出现命令行显示效果不太好的问题。在 + on_train_begin(), on_train_end(), on_exception()中请不要使用该属性，通过print输出即可。""" + return self._trainer.pbar + + @property + def update_every(self): + """Trainer中的模型多少次反向传播才进行一次梯度更新，在Trainer初始化时传入的。""" + return self._trainer.update_every + + @property + def batch_per_epoch(self): + """每个epoch一共有多少个batch，只有在on_epoch_begin之后才能调用该属性。""" + return self._trainer.batch_per_epoch + + def on_train_begin(self): + """ + 在Train过程开始之前调用。 - def before_train(self): - # before the main training loop + :return: + """ pass + + def on_epoch_begin(self): + """ + 在每个epoch开始之前调用一次 - def before_epoch(self, cur_epoch, total_epoch): - # at the beginning of each epoch + :return: + """ pass + + def on_batch_begin(self, batch_x, batch_y, indices): + """ + 每次采集到一个batch的数据则调用一次。这里对batch_x或batch_y删除添加内容是可以影响到Trainer中内容的。所以在这一步 + 可以进行一些负采样之类的操作 - def before_batch(self, batch_x, batch_y, indices): - # at the beginning of each step/mini-batch + :param dict batch_x: DataSet中被设置为input的field的batch。 + :param dict batch_y: DataSet中被设置为target的field的batch。 + :param list(int) indices: 这次采样使用到的indices，可以通过DataSet[indices]获取出这个batch采出的Instance，在一些 + 情况下可以帮助定位是哪个Sample导致了错误。仅在Trainer的prefetch为False时可用。 + :return: + """ pass + + def on_loss_begin(self, batch_y, predict_y): + """ + 在计算loss前调用，即这里修改batch_y或predict_y的值是可以影响到loss计算的。 - def before_loss(self, batch_y, predict_y): - # after data_forward, and before loss computation + :param dict batch_y: 在DataSet中被设置为target的field的batch集合。 + :param dict predict_y: 模型的forward()返回的结果。 + :return: + """ pass + + def on_backward_begin(self, loss): + """ + 在loss得到之后，但在反向传播之前。可能可以进行loss是否为NaN的检查。 - def before_backward(self, loss, model): - # after loss computation, and before gradient backward + :param torch.Tensor loss: 计算得到的loss值 + :return: + """ pass + + def on_backward_end(self): + """ + 反向梯度传播已完成，但由于update_every的设置，可能并不是每一次调用都有梯度。到这一步，还没有更新参数。 - def after_backward(self, model): + :return: + """ pass + + def on_step_end(self): + """ + 到这里模型的参数已经按照梯度更新。但可能受update_every影响，并不是每次都更新了。 - def after_step(self, optimizer): + :return: + """ pass + + def on_batch_end(self): + """ + 这一步与on_step_end是紧接着的。只是为了对称性加上了这一步。 - def after_batch(self, *args): - # at the end of each step/mini-batch + """ pass - - def after_valid(self, eval_result, metric_key, optimizer): + + def on_valid_begin(self): """ - 每次执行验证机的evaluation后会调用。传入eval_result + 如果Trainer中设置了验证，则发生验证前会调用该函数 - :param eval_result: Dict[str: Dict[str: float]], evaluation的结果 - :param metric_key: str - :param optimizer: :return: """ pass - - def after_epoch(self, cur_epoch, n_epoch, optimizer): + + def on_valid_end(self, eval_result, metric_key, optimizer, is_better_eval): """ - 每个epoch结束将会调用该方法 + 每次执行验证集的evaluation后会调用。 - :param cur_epoch: int, 当前的batch。从1开始。 - :param n_epoch: int, 总的batch数 - :param optimizer: 传入Trainer的optimizer。 + :param Dict[str: Dict[str: float]] eval_result: , evaluation的结果。一个例子为{'AccuracyMetric':{'acc':1.0}}，即 + 传入的dict是有两层，第一层是metric的名称，第二层是metric的具体指标。 + :param str metric_key: 初始化Trainer时传入的metric_key。 + :param torch.Optimizer optimizer: Trainer中使用的优化器。 + :param bool is_better_eval: 当前dev结果是否比之前的好。 :return: """ pass - - def after_train(self, model): + + def on_epoch_end(self): + """ + 每个epoch结束将会调用该方法 + """ + pass + + def on_train_end(self): """ 训练结束，调用该方法 - - :param model: nn.Module, 传入Trainer的模型 - :return: """ pass - - def on_exception(self, exception, model, indices): + + def on_exception(self, exception): """ 当训练过程出现异常，会触发该方法 :param exception: 某种类型的Exception，比如KeyboardInterrupt等 - :param model: 传入Trainer的模型 - :param indices: 当前batch的index - :return: """ pass -def transfer(func): - """装饰器，将对CallbackManager的调用转发到各个Callback子类. +def _transfer(func): + """装饰器，将对CallbackManager的调用转发到各个Callback子类. + :param func: :return: """ - + def wrapper(manager, *arg): returns = [] for callback in manager.callbacks: - for env_name, env_value in manager.env.items(): - setattr(callback, env_name, env_value) returns.append(getattr(callback, func.__name__)(*arg)) return returns - + return wrapper class CallbackManager(Callback): - """A manager for all callbacks passed into Trainer. - It collects resources inside Trainer and raise callbacks. - - """ - def __init__(self, env, callbacks=None): """ + 内部使用的Callback管理类 :param dict env: The key is the name of the Trainer attribute(str). The value is the attribute itself. - :param Callback callbacks: + :param List[Callback] callbacks: """ super(CallbackManager, self).__init__() # set attribute of trainer environment - self.env = env - + self.callbacks = [] if callbacks is not None: if isinstance(callbacks, list): @@ -123,104 +306,88 @@ class CallbackManager(Callback): raise TypeError(f"Expect sub-classes of Callback. Got {type(obj)}") else: raise TypeError(f"Expect callbacks in CallbackManager(callbacks) to be list. Got {type(callbacks)}.") - - @transfer - def before_train(self): + + for env_name, env_val in env.items(): + for callback in self.callbacks: + setattr(callback, '_' + env_name, env_val) # Callback.trainer + + @_transfer + def on_train_begin(self): pass - - @transfer - def before_epoch(self, cur_epoch, total_epoch): + + @_transfer + def on_epoch_begin(self): pass - - @transfer - def before_batch(self, batch_x, batch_y, indices): + + @_transfer + def on_batch_begin(self, batch_x, batch_y, indices): pass - - @transfer - def before_loss(self, batch_y, predict_y): + + @_transfer + def on_loss_begin(self, batch_y, predict_y): pass - - @transfer - def before_backward(self, loss, model): + + @_transfer + def on_backward_begin(self, loss): pass - - @transfer - def after_backward(self, model): + + @_transfer + def on_backward_end(self): pass - - @transfer - def after_step(self, optimizer): + + @_transfer + def on_step_end(self): pass - - @transfer - def after_batch(self): + + @_transfer + def on_batch_end(self): pass - - @transfer - def after_valid(self, eval_result, metric_key, optimizer): + + @_transfer + def on_valid_begin(self): pass - - @transfer - def after_epoch(self, cur_epoch, n_epoch, optimizer): + + @_transfer + def on_valid_end(self, eval_result, metric_key, optimizer, is_better_eval): pass - - @transfer - def after_train(self, model): + + @_transfer + def on_epoch_end(self): pass - - @transfer - def on_exception(self, exception, model, indices): + + @_transfer + def on_train_end(self): + pass + + @_transfer + def on_exception(self, exception): pass -class DummyCallback(Callback): - def before_train(self, *arg): - print(arg) - - def after_epoch(self, cur_epoch, n_epoch, optimizer): - print(cur_epoch, n_epoch, optimizer) - - -class EchoCallback(Callback): - def before_train(self): - print("before_train") - - def before_epoch(self, cur_epoch, total_epoch): - print("before_epoch") - - def before_batch(self, batch_x, batch_y, indices): - print("before_batch") - - def before_loss(self, batch_y, predict_y): - print("before_loss") - - def before_backward(self, loss, model): - print("before_backward") +class GradientClipCallback(Callback): + """ + 别名：:class:`fastNLP.GradientClipCallback` :class:`fastNLP.core.callback.GradientClipCallback` - def after_batch(self): - print("after_batch") + 每次backward前，将parameter的gradient clip到某个范围。 - def after_epoch(self, cur_epoch, n_epoch, optimizer): - print("after_epoch") + :param None,torch.Tensor,List[torch.Tensor] parameters: 一般通过model.parameters()获得。 + 如果为None则默认对Trainer的model中所有参数进行clip + :param float clip_value: 将gradient 限制到[-clip_value, clip_value]。clip_value应该为正数 + :param str clip_type: 支持'norm', 'value' + 两种:: - def after_train(self, model): - print("after_train") + 1 'norm', 将gradient的norm rescale到[-clip_value, clip_value] + + 2 'value', 将gradient限制在[-clip_value, clip_value], + 小于-clip_value的gradient被赋值为-clip_value; + 大于clip_value的gradient被赋值为clip_value. -class GradientClipCallback(Callback): + """ + def __init__(self, parameters=None, clip_value=1, clip_type='norm'): - """ - 每次backward前，将parameter的gradient clip到某个范围。 - - :param parameters: None, torch.Tensor或List[torch.Tensor], 一般通过model.parameters()获得。如果为None则默认对Trainer - 的model中所有参数进行clip - :param clip_value: float, 将gradient 限制到[-clip_value, clip_value]。clip_value应该为正数 - :param clip_type: str, 支持'norm', 'value'两种。 - (1) 'norm', 将gradient的norm rescale到[-clip_value, clip_value] - (2) 'value', 将gradient限制在[-clip_value, clip_value], 小于-clip_value的gradient被赋值为-clip_value; 大于 - clip_value的gradient被赋值为clip_value. - """ + super().__init__() - + from torch import nn if clip_type == 'norm': self.clip_fun = nn.utils.clip_grad_norm_ @@ -230,13 +397,352 @@ class GradientClipCallback(Callback): raise ValueError("Only supports `norm` or `value` right now.") self.parameters = parameters self.clip_value = clip_value + + def on_backward_end(self): + if self.parameters is None: + self.clip_fun(self.model.parameters(), self.clip_value) + else: + self.clip_fun(self.parameters, self.clip_value) - def after_backward(self, model): - self.clip_fun(model.parameters(), self.clip_value) +class EarlyStopCallback(Callback): + """ + 别名：:class:`fastNLP.EarlyStopCallback` :class:`fastNLP.core.callback.EarlyStopCallback` + + 多少个epoch没有变好就停止训练，相关类 :class:`EarlyStopError` + + :param int patience: epoch的数量 + """ + + def __init__(self, patience): + super(EarlyStopCallback, self).__init__() + self.patience = patience + self.wait = 0 + + def on_valid_end(self, eval_result, metric_key, optimizer, is_better_eval): + if not is_better_eval: + # current result is getting worse + if self.wait == self.patience: + raise EarlyStopError("Early stopping raised.") + else: + self.wait += 1 + else: + self.wait = 0 + + def on_exception(self, exception): + if isinstance(exception, EarlyStopError): + print("Early Stopping triggered in epoch {}!".format(self.epoch)) + else: + raise exception # 抛出陌生Error -if __name__ == "__main__": - manager = CallbackManager(env={"n_epoch": 3}, callbacks=[DummyCallback(), DummyCallback()]) - manager.before_train(10, 11, 12) - # print(manager.after_epoch()) +class FitlogCallback(Callback): + """ + 该callback将loss和progress自动写入到fitlog中; 如果Trainer有dev的数据，将自动把dev的结果写入到log中; 同时还支持传入 + 一个(或多个)test数据集进行测试(只有在trainer具有dev时才能使用)，每次在dev上evaluate之后会在这些数据集上验证一下。 + 并将验证结果写入到fitlog中。这些数据集的结果是根据dev上最好的结果报道的，即如果dev在第3个epoch取得了最佳，则 + fitlog中记录的关于这些数据集的结果就是来自第三个epoch的结果。 + + :param DataSet,dict(DataSet) data: 传入DataSet对象，会使用多个Trainer中的metric对数据进行验证。如果需要传入多个 + DataSet请通过dict的方式传入，dict的key将作为对应dataset的name传递给fitlog。若tester不为None时，data需要通过 + dict的方式传入。如果仅传入DataSet, 则被命名为test + :param Tester tester: Tester对象，将在on_valid_end时调用。tester中的DataSet会被称为为`test` + :param int verbose: 是否在终端打印内容，0不打印 + :param bool log_exception: fitlog是否记录发生的exception信息 + """ + # 还没有被导出到 fastNLP 层 + # 别名: :class:`fastNLP.FitlogCallback` :class:`fastNLP.core.callback.FitlogCallback` + + def __init__(self, data=None, tester=None, verbose=0, log_exception=False): + super().__init__() + self.datasets = {} + self.testers = {} + self._log_exception = log_exception + if tester is not None: + assert isinstance(tester, Tester), "Only fastNLP.Tester allowed." + assert isinstance(data, dict) or data is None, "If tester is not None, only dict[DataSet] allowed for data." + if data is not None: + assert 'test' not in data, "Cannot use `test` as DataSet key, when tester is passed." + setattr(tester, 'verbose', 0) + self.testers['test'] = tester + + if isinstance(data, dict): + for key, value in data.items(): + assert isinstance(value, DataSet), f"Only DataSet object is allowed, not {type(value)}." + for key, value in data.items(): + self.datasets[key] = value + elif isinstance(data, DataSet): + self.datasets['test'] = data + else: + raise TypeError("data receives dict[DataSet] or DataSet object.") + + self.verbose = verbose + + def on_train_begin(self): + if (len(self.datasets) > 0 or len(self.testers) > 0) and self.trainer.dev_data is None: + raise RuntimeError("Trainer has no dev data, you cannot pass extra data to do evaluation.") + + if len(self.datasets) > 0: + for key, data in self.datasets.items(): + tester = Tester(data=data, model=self.model, batch_size=self.batch_size, metrics=self.trainer.metrics, + verbose=0) + self.testers[key] = tester + fitlog.add_progress(total_steps=self.n_steps) + + def on_backward_begin(self, loss): + fitlog.add_loss(loss.item(), name='loss', step=self.step, epoch=self.epoch) + + def on_valid_end(self, eval_result, metric_key, optimizer, better_result): + if better_result: + eval_result = deepcopy(eval_result) + eval_result['step'] = self.step + eval_result['epoch'] = self.epoch + fitlog.add_best_metric(eval_result) + fitlog.add_metric(eval_result, step=self.step, epoch=self.epoch) + if len(self.testers) > 0: + for key, tester in self.testers.items(): + try: + eval_result = tester.test() + if self.verbose != 0: + self.pbar.write("Evaluation on DataSet {}:".format(key)) + self.pbar.write(tester._format_eval_results(eval_result)) + fitlog.add_metric(eval_result, name=key, step=self.step, epoch=self.epoch) + if better_result: + fitlog.add_best_metric(eval_result, name=key) + except Exception: + self.pbar.write("Exception happens when evaluate on DataSet named `{}`.".format(key)) + + def on_train_end(self): + fitlog.finish() + + def on_exception(self, exception): + fitlog.finish(status=1) + if self._log_exception: + fitlog.add_other(str(exception), name='except_info') + + +class LRScheduler(Callback): + """ + 别名：:class:`fastNLP.LRScheduler` :class:`fastNLP.core.callback.LRScheduler` + + 对PyTorch LR Scheduler的包装以使得其可以被Trainer所使用 + + :param torch.optim.lr_scheduler._LRScheduler lr_scheduler: PyTorch的lr_scheduler + """ + + def __init__(self, lr_scheduler): + + super(LRScheduler, self).__init__() + import torch.optim + if isinstance(lr_scheduler, torch.optim.lr_scheduler._LRScheduler): + self.scheduler = lr_scheduler + else: + raise ValueError(f"Expect torch.optim.lr_scheduler for LRScheduler. Got {type(lr_scheduler)}.") + + def on_epoch_begin(self): + self.scheduler.step(self.epoch) + + +class ControlC(Callback): + """ + 别名：:class:`fastNLP.ControlC` :class:`fastNLP.core.callback.ControlC` + + :param bool quit_all: 若为True,则检测到control+C 直接退出程序；否则只退出Trainer + """ + + def __init__(self, quit_all): + + super(ControlC, self).__init__() + if type(quit_all) != bool: + raise ValueError("In KeyBoardInterrupt, quit_all arguemnt must be a bool.") + self.quit_all = quit_all + + def on_exception(self, exception): + if isinstance(exception, KeyboardInterrupt): + if self.quit_all is True: + import sys + sys.exit(0) # 直接退出程序 + else: + pass + else: + raise exception # 抛出陌生Error + + +class SmoothValue(object): + def __init__(self, beta: float): + self.beta, self.n, self.mov_avg = beta, 0, 0 + self.smooth = None + + def add_value(self, val: float) -> None: + "Add `val` to calculate updated smoothed value." + self.n += 1 + self.mov_avg = self.beta * self.mov_avg + (1 - self.beta) * val + self.smooth = self.mov_avg / (1 - self.beta ** self.n) + + +class LRFinder(Callback): + """ + 别名：:class:`fastNLP.LRFinder` :class:`fastNLP.core.callback.LRFinder` + + 用第一个 epoch 找最佳的学习率，从第二个epoch开始应用它 + + :param float start_lr: 学习率下界 + :param float end_lr: 学习率上界 + """ + + def __init__(self, start_lr=1e-6, end_lr=10): + + super(LRFinder, self).__init__() + self.start_lr, self.end_lr = start_lr, end_lr + + self.stop = False + self.best_loss = 0. + self.best_lr = None + self.loss_history = [] + self.smooth_value = SmoothValue(0.8) + self.opt = None + self.find = None + self.loader = ModelLoader() + + @property + def lr_gen(self): + scale = (self.end_lr - self.start_lr) / self.batch_per_epoch + return (self.start_lr + scale * (step + 1) for step in range(self.batch_per_epoch)) + + @property + def num_it(self): + return self.batch_per_epoch + + def on_epoch_begin(self): + if self.epoch == 1: # first epoch + self.opt = self.trainer.optimizer # pytorch optimizer + self.opt.param_groups[0]["lr"] = self.start_lr + # save model + ModelSaver("tmp").save_pytorch(self.trainer.model, param_only=True) + self.find = True + + def on_backward_begin(self, loss): + if self.find: + if torch.isnan(loss) or self.stop is True: + self.stop = True + return + loss_val = loss.detach().mean().item() + self.loss_history.append(loss_val) + self.smooth_value.add_value(loss_val) + if self.best_loss == 0. or self.smooth_value.smooth < self.best_loss: + self.best_loss = self.smooth_value.smooth + self.best_lr = self.opt.param_groups[0]["lr"] + + def on_batch_end(self, *args): + if self.find: + lr = next(self.lr_gen, None) + if lr is None or self.stop is True or self.loss_history[-1] > 4 * self.best_loss: + self.stop = True + return + self.opt.param_groups[0]["lr"] = lr + # self.loader.load_pytorch(self.trainer.model, "tmp") + + def on_epoch_end(self): + if self.epoch == 1: # first epoch + self.opt.param_groups[0]["lr"] = self.best_lr + self.find = False + # reset model + ModelLoader().load_pytorch(self.trainer.model, "tmp") + self.pbar.write("Model reset. \nFind best lr={}".format(self.best_lr)) + + +class TensorboardCallback(Callback): + """ + 别名：:class:`fastNLP.TensorboardCallback` :class:`fastNLP.core.callback.TensorboardCallback` + + 接受以下一个或多个字符串作为参数： + - "model" + - "loss" + - "metric" + + .. warning:: + fastNLP 已停止对此功能的维护，请等待 fastNLP 兼容 PyTorch1.1 的下一个版本。 + 或者使用和 fastNLP 高度配合的 fitlog（参见 :doc:`/user/with_fitlog` ）。 + + """ + + def __init__(self, *options): + super(TensorboardCallback, self).__init__() + args = {"model", "loss", "metric"} + for opt in options: + if opt not in args: + raise ValueError("Unrecognized argument {}. Expect one of {}".format(opt, args)) + self.options = options + self._summary_writer = None + self.graph_added = False + + def on_train_begin(self): + save_dir = self.trainer.save_path + if save_dir is None: + path = os.path.join("./", 'tensorboard_logs_{}'.format(self.trainer.start_time)) + else: + path = os.path.join(save_dir, 'tensorboard_logs_{}'.format(self.trainer.start_time)) + if tensorboardX_flag: + self._summary_writer = SummaryWriter(path) + else: + self._summary_writer = None + + def on_batch_begin(self, batch_x, batch_y, indices): + if "model" in self.options and self.graph_added is False: + # tesorboardX 这里有大bug，暂时没法画模型图 + # from fastNLP.core.utils import _build_args + # inputs = _build_args(self.trainer.model, **batch_x) + # args = tuple([value for value in inputs.values()]) + # args = args[0] if len(args) == 1 else args + # self._summary_writer.add_graph(self.trainer.model, torch.zeros(32, 2)) + self.graph_added = True + + def on_backward_begin(self, loss): + if "loss" in self.options and self._summary_writer: + self._summary_writer.add_scalar("loss", loss.item(), global_step=self.trainer.step) + + if "model" in self.options and self._summary_writer: + for name, param in self.trainer.model.named_parameters(): + if param.requires_grad: + self._summary_writer.add_scalar(name + "_mean", param.mean(), global_step=self.trainer.step) + # self._summary_writer.add_scalar(name + "_std", param.std(), global_step=self.trainer.step) + self._summary_writer.add_scalar(name + "_grad_mean", param.grad.mean(), + global_step=self.trainer.step) + + def on_valid_end(self, eval_result, metric_key, optimizer, is_better_eval): + if "metric" in self.options and self._summary_writer: + for name, metric in eval_result.items(): + for metric_key, metric_val in metric.items(): + self._summary_writer.add_scalar("valid_{}_{}".format(name, metric_key), metric_val, + global_step=self.trainer.step) + + def on_train_end(self): + if self._summary_writer: + self._summary_writer.close() + del self._summary_writer + + def on_exception(self, exception): + if hasattr(self, "_summary_writer"): + self._summary_writer.close() + del self._summary_writer + + +class CallbackException(BaseException): + """ + 当需要通过callback跳出训练的时候可以通过抛出CallbackException并在on_exception中捕获这个值。 + + :param str msg: Exception的信息。 + """ + + def __init__(self, msg): + super(CallbackException, self).__init__(msg) + + +class EarlyStopError(CallbackException): + """ + 用于EarlyStop时从Trainer训练循环中跳出。 + + """ + + def __init__(self, msg): + super(EarlyStopError, self).__init__(msg) diff --git a/fastNLP/core/const.py b/fastNLP/core/const.py new file mode 100644 index 00000000..89ff51a2 --- /dev/null +++ b/fastNLP/core/const.py @@ -0,0 +1,59 @@ +class Const: + """ + fastNLP中field命名常量。 + + .. todo:: + 把下面这段改成表格 + + 具体列表:: + + INPUT 模型的序列输入 words（复数words1, words2） + CHAR_INPUT 模型character输入 chars（复数chars1， chars2） + INPUT_LEN 序列长度 seq_len（复数seq_len1，seq_len2） + OUTPUT 模型输出 pred（复数pred1， pred2） + TARGET 真实目标 target（复数target1，target2） + LOSS 损失函数 loss （复数loss1，loss2） + + """ + INPUT = 'words' + CHAR_INPUT = 'chars' + INPUT_LEN = 'seq_len' + OUTPUT = 'pred' + TARGET = 'target' + LOSS = 'loss' + + @staticmethod + def INPUTS(i): + """得到第 i 个 ``INPUT`` 的命名""" + i = int(i) + 1 + return Const.INPUT + str(i) + + @staticmethod + def CHAR_INPUTS(i): + """得到第 i 个 ``CHAR_INPUT`` 的命名""" + i = int(i) + 1 + return Const.CHAR_INPUT + str(i) + + @staticmethod + def INPUT_LENS(i): + """得到第 i 个 ``INPUT_LEN`` 的命名""" + i = int(i) + 1 + return Const.INPUT_LEN + str(i) + + @staticmethod + def OUTPUTS(i): + """得到第 i 个 ``OUTPUT`` 的命名""" + i = int(i) + 1 + return Const.OUTPUT + str(i) + + @staticmethod + def TARGETS(i): + """得到第 i 个 ``TARGET`` 的命名""" + i = int(i) + 1 + return Const.TARGET + str(i) + + @staticmethod + def LOSSES(i): + """得到第 i 个 ``LOSS`` 的命名""" + i = int(i) + 1 + return Const.LOSS + str(i) diff --git a/fastNLP/core/dataset.py b/fastNLP/core/dataset.py index 64aa2934..9f24adf2 100644 --- a/fastNLP/core/dataset.py +++ b/fastNLP/core/dataset.py @@ -1,28 +1,304 @@ +""" +:class:`~fastNLP.core.dataset.DataSet` 是fastNLP中用于承载数据的容器。可以将DataSet看做是一个表格， +每一行是一个sample (在fastNLP中被称为 :mod:`~.instance` )， +每一列是一个feature (在fastNLP中称为 :mod:`.field` )。 + +.. csv-table:: Following is a demo layout of DataSet + :header: "sentence", "words", "seq_len" + + "This is the first instance .", "[This, is, the, first, instance, .]", 6 + "Second instance .", "[Second, instance, .]", 3 + "Third instance .", "[Third, instance, .]", 3 + "...", "[...]", "..." + +在fastNLP内部每一行是一个 :class:`~fastNLP.Instance` 对象； 每一列是一个 :class:`~fastNLP.FieldArray` 对象。 + +1 DataSet的创建 + 创建DataSet主要有以下的3种方式 + +1.1 传入dict + + Example:: + + from fastNLP import DataSet + data = {'sentence':["This is the first instance .", "Second instance .", "Third instance ."], + 'words': [['this', 'is', 'the', 'first', 'instance', '.'], ['Second', 'instance', '.'], ['Third', 'instance', '.'], + 'seq_len': [6, 3, 3]} + dataset = DataSet(data) + # 传入的dict的每个key的value应该为具有相同长度的list + +1.2 通过构建Instance + + Example:: + + from fastNLP import DataSet + from fastNLP import Instance + dataset = DataSet() + instance = Instance(sentence="This is the first instance", + words=['this', 'is', 'the', 'first', 'instance', '.'], + seq_len=6) + dataset.append(instance) + # 可以继续append更多内容，但是append的instance应该和第一个instance拥有完全相同的field + +1.3 通过list(Instance) + + Example:: + + from fastNLP import DataSet + from fastNLP import Instance + instances = [] + instances.append(Instance(sentence="This is the first instance", + words=['this', 'is', 'the', 'first', 'instance', '.'], + seq_len=6)) + instances.append(Instance(sentence="Second instance .", + words=['Second', 'instance', '.'], + seq_len=3)) + dataset = DataSet(instances) + +2 DataSet与预处理 + 常见的预处理有如下几种 + +2.1 从某个文本文件读取内容 # + + .. todo:: + 引用DataLoader + + Example:: + + from fastNLP import DataSet + from fastNLP import Instance + dataset = DataSet() + filepath='some/text/file' + # 假设文件中每行内容如下(sentence label): + # This is a fantastic day positive + # The bad weather negative + # ..... + with open(filepath, 'r') as f: + for line in f: + sent, label = line.strip().split('\t') + dataset.append(Instance(sentence=sent, label=label)) + +2.2 index, 返回结果为对DataSet对象的浅拷贝 + + Example:: + + import numpy as np + from fastNLP import DataSet + dataset = DataSet({'a': np.arange(10), 'b': [[_] for _ in range(10)]}) + d[0] # 使用一个下标获取一个instance + >>{'a': 0 type=int,'b': [2] type=list} # 得到一个instance + d[1:3] # 使用slice获取一个新的DataSet + >>DataSet({'a': 1 type=int, 'b': [2] type=list}, {'a': 2 type=int, 'b': [2] type=list}) + +2.3 对DataSet中的内容处理 + + Example:: + + from fastNLP import DataSet + data = {'sentence':["This is the first instance .", "Second instance .", "Third instance ."]} + dataset = DataSet(data) + # 将句子分成单词形式, 详见DataSet.apply()方法 + dataset.apply(lambda ins: ins['sentence'].split(), new_field_name='words') + # 或使用DataSet.apply_field() + dataset.apply_field(lambda sent:sent.split(), field_name='sentence', new_field_name='words') + # 除了匿名函数，也可以定义函数传递进去 + def get_words(instance): + sentence = instance['sentence'] + words = sentence.split() + return words + dataset.apply(get_words, new_field_name='words') + +2.4 删除DataSet的内容 + + Example:: + + from fastNLP import DataSet + dataset = DataSet({'a': list(range(-5, 5))}) + # 返回满足条件的instance,并放入DataSet中 + dropped_dataset = dataset.drop(lambda ins:ins['a']<0, inplace=False) + # 在dataset中删除满足条件的instance + dataset.drop(lambda ins:ins['a']<0) # dataset的instance数量减少 + # 删除第3个instance + dataset.delete_instance(2) + # 删除名为'a'的field + dataset.delete_field('a') + + +2.5 遍历DataSet的内容 + + Example:: + + for instance in dataset: + # do something + +2.6 一些其它操作 + + Example:: + + # 检查是否存在名为'a'的field + dataset.has_field('a') # 或 ('a' in dataset) + # 将名为'a'的field改名为'b' + dataset.rename_field('a', 'b') + # DataSet的长度 + len(dataset) + +3 DataSet与自然语言处理(NLP) + 在目前深度学习的模型中，大都依赖于随机梯度下降法(SGD)进行模型的优化。随机梯度下降需要将数据切分成一个一个的Batch， + 一个Batch进行一次前向计算(forward)与梯度后向传播(backward)。在自然语言处理的场景下，往往还需要对数据进行pad。这是 + 由于句子的长度一般是不同的，但是一次Batch中的每个field都必须是一个tensor，所以需要将所有句子都补齐到相同的长度。 + +3.1 DataSet与Batch + + 我们先看fastNLP中如何将数据分成一个一个的Batch的例子, 这里我们使用随机生成的数据来模拟一个二分类文本分类任务， + words和characters是输入，labels是文本类别 + + Example:: + + from fastNLP import DataSet + from fastNLP import Batch + from fastNLP import SequentialSampler + from fastNLP import EngChar2DPadder + + num_instances = 100 + # 假设每句话最少2个词，最多5个词; 词表的大小是100个; 一共26个字母，每个单词最短1个字母，最长5个字母 + lengths = [random.randint(2, 5) for _ in range(num_instances)] + data = {'words': [[random.randint(1, 100) for _ in range(lengths[idx]) ] for idx in range(num_instances)], + 'chars': [ + [[random.randint(1, 27) for _ in range(random.randint(1, 5))] + for _ in range(lengths[idx])] + for idx in range(num_instances)], + 'label': [random.randint(0, 1) for _ in range(num_instances)]} + + d = DataSet(data) + d.set_padder('chars', EngChar2DPadder()) # 因为英文character的pad方式与word的pad方式不一样 + + d.set_target('label') + d.set_input('words', 'chars') + + for batch_x, batch_y in Batch(d, sampler=SequentialSampler(), batch_size=2): + print("batch_x:", batch_x) + print("batch_y:", batch_y) + break + # 输出为 + # {'words': tensor([[49, 27, 20, 36, 63], + # [53, 82, 23, 11, 0]]), 'chars': tensor([[[13, 3, 14, 25, 1], + # [ 8, 20, 12, 0, 0], + # [27, 8, 0, 0, 0], + # [ 1, 15, 26, 0, 0], + # [11, 24, 17, 0, 0]], + # + # [[ 6, 14, 11, 27, 22], + # [18, 6, 4, 19, 0], + # [19, 22, 9, 0, 0], + # [10, 25, 0, 0, 0], + # [ 0, 0, 0, 0, 0]]])} + # {'label': tensor([0, 0])} + + 其中 :class:`~fastNLP.Batch` 是用于从DataSet中按照batch_size为大小取出batch的迭代器， + :class:`~fastNLP.SequentialSampler` 用于指示 Batch 以怎样的 + 顺序从DataSet中取出instance以组成一个batch， + 更详细的说明请参照 :class:`~fastNLP.Batch` 和 :class:`~fastNLP.SequentialSampler` 文档。 + + 通过DataSet.set_input('words', 'chars'), fastNLP将认为'words'和'chars'这两个field都是input，并将它们都放入迭代器 + 生成的第一个dict中; DataSet.set_target('labels'), fastNLP将认为'labels'这个field是target，并将其放入到迭代器的第 + 二个dict中。如上例中所打印结果。分为input和target的原因是由于它们在被 :class:`~fastNLP.Trainer` 所使用时会有所差异， + 详见 :class:`~fastNLP.Trainer` + + 当把某个field设置为'target'或者'input'的时候(两者不是互斥的，可以同时设为input和target)，fastNLP不仅仅只是将其放 + 置到不同的dict中，而还会对被设置为input或target的field进行类型检查。类型检查的目的是为了看能否把该field转为 + pytorch的torch.LongTensor或torch.FloatTensor类型(也可以在Batch中设置输出numpy类型，参考 :class:`~fastNLP.Batch` )，如上例所示， + fastNLP已将words，chars和label转为了Tensor类型。如果field在每个instance都拥有相同的维度(不能超过两维)，且最内层 + 的元素都为相同的type(int, float, np.int*, np.float*)，则fastNLP默认将对该field进行pad。也支持全为str的field作为 + target和input，这种情况下，fastNLP默认不进行pad。另外，当某个field已经被设置为了target或者input后，之后append的 + instance对应的field必须要和前面已有的内容一致，否则会报错。 + + 可以查看field的dtype:: + + from fastNLP import DataSet + + d = DataSet({'a': [0, 1, 3], 'b':[[1.0, 2.0], [0.1, 0.2], [3]]}) + d.set_input('a', 'b') + d.a.dtype + >> numpy.int64 + d.b.dtype + >> numpy.float64 + # 默认情况下'a'这个field将被转换为torch.LongTensor，但如果需要其为torch.FloatTensor可以手动修改dtype + d.a.dtype = float # 请确保该field的确可以全部转换为float。 + + 如果某个field中出现了多种类型混合(比如一部分为str，一部分为int)的情况，fastNLP无法判断该field的类型，会报如下的 + 错误:: + + from fastNLP import DataSet + d = DataSet({'data': [1, 'a']}) + d.set_input('data') + >> RuntimeError: Mixed data types in Field data: [, ] + + 可以通过设置以忽略对该field进行类型检查:: + + from fastNLP import DataSet + d = DataSet({'data': [1, 'a']}) + d.set_ignore_type('data') + d.set_input('data') + + 当某个field被设置为忽略type之后，fastNLP将不对其进行pad。 + +3.2 DataSet与pad + + 在fastNLP里，pad是与一个field绑定的。即不同的field可以使用不同的pad方式，比如在英文任务中word需要的pad和 + character的pad方式往往是不同的。fastNLP是通过一个叫做 :class:`~fastNLP.Padder` 的子类来完成的。 + 默认情况下，所有field使用 :class:`~fastNLP.AutoPadder` + 。可以通过使用以下方式设置Padder(如果将padder设置为None，则该field不会进行pad操作)。 + 大多数情况下直接使用 :class:`~fastNLP.AutoPadder` 就可以了。 + 如果 :class:`~fastNLP.AutoPadder` 或 :class:`~fastNLP.EngChar2DPadder` 无法满足需求， + 也可以自己写一个 :class:`~fastNLP.Padder` 。 + + Example:: + + from fastNLP import DataSet + from fastNLP import EngChar2DPadder + import random + dataset = DataSet() + max_chars, max_words, sent_num = 5, 10, 20 + contents = [[ + [random.randint(1, 27) for _ in range(random.randint(1, max_chars))] + for _ in range(random.randint(1, max_words)) + ] for _ in range(sent_num)] + # 初始化时传入 + dataset.add_field('chars', contents, padder=EngChar2DPadder()) + # 直接设置 + dataset.set_padder('chars', EngChar2DPadder()) + # 也可以设置pad的value + dataset.set_pad_val('chars', -1) + + +""" +__all__ = [ + "DataSet" +] + import _pickle as pickle +import warnings import numpy as np -from fastNLP.core.fieldarray import FieldArray -from fastNLP.core.instance import Instance -from fastNLP.core.utils import get_func_signature -from fastNLP.io.base_loader import DataLoaderRegister +from .field import AutoPadder +from .field import FieldArray +from .instance import Instance +from .utils import _get_func_signature class DataSet(object): - """DataSet is the collection of examples. - DataSet provides instance-level interface. You can append and access an instance of the DataSet. - However, it stores data in a different way: Field-first, Instance-second. - """ + 别名：:class:`fastNLP.DataSet` :class:`fastNLP.core.dataset.DataSet` - def __init__(self, data=None): - """ + fastNLP的数据容器，详细的使用方法见文档 :doc:`fastNLP.core.dataset` + + :param data: 如果为dict类型，则每个key的value应该为等长的list; 如果为list， + 每个元素应该为具有相同field的 :class:`~fastNLP.Instance` 。 - :param data: a dict or a list. - If `data` is a dict, the key is the name of a FieldArray and the value is the FieldArray. All values - must be of the same length. - If `data` is a list, it must be a list of Instance objects. - """ + """ + + def __init__(self, data=None): self.field_arrays = {} if data is not None: if isinstance(data, dict): @@ -31,51 +307,48 @@ class DataSet(object): length_set.add(len(value)) assert len(length_set) == 1, "Arrays must all be same length." for key, value in data.items(): - self.add_field(name=key, fields=value) + self.add_field(field_name=key, fields=value) elif isinstance(data, list): for ins in data: assert isinstance(ins, Instance), "Must be Instance type, not {}.".format(type(ins)) self.append(ins) - + else: raise ValueError("data only be dict or list type.") - + def __contains__(self, item): return item in self.field_arrays - + def __iter__(self): def iter_func(): for idx in range(len(self)): yield self[idx] - + return iter_func() - + def _inner_iter(self): class Iter_ptr: def __init__(self, dataset, idx): self.dataset = dataset self.idx = idx - + def __getitem__(self, item): assert item in self.dataset.field_arrays, "no such field:{} in Instance {}".format(item, self.dataset[ self.idx]) assert self.idx < len(self.dataset.field_arrays[item]), "index:{} out of range".format(self.idx) return self.dataset.field_arrays[item][self.idx] - + def __repr__(self): return self.dataset[self.idx].__repr__() - + def inner_iter_func(): for idx in range(len(self)): yield Iter_ptr(self, idx) - + return inner_iter_func() - + def __getitem__(self, idx): - """Fetch Instance(s) at the `idx` position(s) in the dataset. - Notice: This method returns a copy of the actual instance(s). Any change to the returned value would not modify - the origin instance(s) of the DataSet. - If you want to make in-place changes to all Instances, use `apply` method. + """给定int的index，返回一个Instance; 给定slice，返回包含这个slice内容的新的DataSet。 :param idx: can be int or slice. :return: If `idx` is int, return an Instance object. @@ -85,36 +358,41 @@ class DataSet(object): return Instance(**{name: self.field_arrays[name][idx] for name in self.field_arrays}) elif isinstance(idx, slice): if idx.start is not None and (idx.start >= len(self) or idx.start <= -len(self)): - raise RuntimeError(f"Start index {idx.start} out of range 0-{len(self)-1}") + raise RuntimeError(f"Start index {idx.start} out of range 0-{len(self) - 1}") data_set = DataSet() for field in self.field_arrays.values(): - data_set.add_field(name=field.name, - fields=field.content[idx], - padding_val=field.padding_val, - is_input=field.is_input, - is_target=field.is_target) + data_set.add_field(field_name=field.name, fields=field.content[idx], padder=field.padder, + is_input=field.is_input, is_target=field.is_target, ignore_type=field.ignore_type) return data_set + elif isinstance(idx, str): + if idx not in self: + raise KeyError("No such field called {} in DataSet.".format(idx)) + return self.field_arrays[idx] + elif isinstance(idx, list): + dataset = DataSet() + for i in idx: + assert isinstance(i, int), "Only int index allowed." + instance = self[i] + dataset.append(instance) + for field_name, field in self.field_arrays.items(): + dataset.field_arrays[field_name].to(field) + return dataset else: raise KeyError("Unrecognized type {} for idx in __getitem__ method".format(type(idx))) - + def __getattr__(self, item): # Not tested. Don't use !! if item == "field_arrays": raise AttributeError if isinstance(item, str) and item in self.field_arrays: return self.field_arrays[item] - try: - reader = DataLoaderRegister.get_reader(item) - return reader - except AttributeError: - raise - + def __setstate__(self, state): self.__dict__ = state - + def __getstate__(self): return self.__dict__ - + def __len__(self): """Fetch the length of the dataset. @@ -124,185 +402,390 @@ class DataSet(object): return 0 field = iter(self.field_arrays.values()).__next__() return len(field) - + def __inner_repr__(self): if len(self) < 20: return ",\n".join([ins.__repr__() for ins in self]) else: return self[:5].__inner_repr__() + "\n...\n" + self[-5:].__inner_repr__() - + def __repr__(self): return "DataSet(" + self.__inner_repr__() + ")" + + def append(self, instance): + """ + 将一个instance对象append到DataSet后面。 - def append(self, ins): - """Add an instance to the DataSet. - If the DataSet is not empty, the instance must have the same field names as the rest instances in the DataSet. - - :param ins: an Instance object + :param instance: :class:`~fastNLP.Instance` 类型。若DataSet不为空，则instance应该拥有和DataSet完全一样的field。 """ if len(self.field_arrays) == 0: # DataSet has no field yet - for name, field in ins.fields.items(): - self.field_arrays[name] = FieldArray(name, [field]) + for name, field in instance.fields.items(): + field = field.tolist() if isinstance(field, np.ndarray) else field + self.field_arrays[name] = FieldArray(name, [field]) # 第一个样本，必须用list包装起来 else: - assert len(self.field_arrays) == len(ins.fields) - for name, field in ins.fields.items(): + if len(self.field_arrays) != len(instance.fields): + raise ValueError( + "DataSet object has {} fields, but attempt to append an Instance object with {} fields." + .format(len(self.field_arrays), len(instance.fields))) + for name, field in instance.fields.items(): assert name in self.field_arrays self.field_arrays[name].append(field) + + def add_fieldarray(self, field_name, fieldarray): + """ + 将fieldarray添加到DataSet中. - def add_field(self, name, fields, padding_val=0, is_input=False, is_target=False): - """Add a new field to the DataSet. + :param str field_name: 新加入的field的名称 + :param fieldarray: :class:`~fastNLP.FieldArray` 类型。需要加入DataSet的field的内容 + :return: + """ + if not isinstance(fieldarray, FieldArray): + raise TypeError("Only fastNLP.FieldArray supported.") + if len(self) != len(fieldarray): + raise RuntimeError(f"The field to add must have the same size as dataset. " + f"Dataset size {len(self)} != field size {len(fieldarray)}") + self.field_arrays[field_name] = fieldarray + + def add_field(self, field_name, fields, padder=AutoPadder(), is_input=False, is_target=False, ignore_type=False): + """ + 新增一个field - :param str name: the name of the field. - :param fields: a list of int, float, or other objects. - :param int padding_val: integer for padding. - :param bool is_input: whether this field is model input. - :param bool is_target: whether this field is label or target. + :param str field_name: 新增的field的名称 + :param list fields: 需要新增的field的内容 + :param None, padder: :class:`~fastNLP.Padder` 类型， + 如果为None,则不进行pad，默认使用 :class:`~fastNLP.AutoPadder` 自动判断是否需要做pad。 + :param bool is_input: 新加入的field是否是input + :param bool is_target: 新加入的field是否是target + :param bool ignore_type: 是否忽略对新加入的field的类型检查 """ + if len(self.field_arrays) != 0: if len(self) != len(fields): - raise RuntimeError(f"The field to append must have the same size as dataset. " + raise RuntimeError(f"The field to add must have the same size as dataset. " f"Dataset size {len(self)} != field size {len(fields)}") - self.field_arrays[name] = FieldArray(name, fields, padding_val=padding_val, is_target=is_target, - is_input=is_input) + self.field_arrays[field_name] = FieldArray(field_name, fields, is_target=is_target, is_input=is_input, + padder=padder, ignore_type=ignore_type) + + def delete_instance(self, index): + """ + 删除第index个instance - def delete_field(self, name): - """Delete a field based on the field name. + :param int index: 需要删除的instance的index，从0开始 + """ + assert isinstance(index, int), "Only integer supported." + if len(self) <= index: + raise IndexError("{} is too large for as DataSet with {} instances.".format(index, len(self))) + if len(self) == 1: + self.field_arrays.clear() + else: + for field in self.field_arrays.values(): + field.pop(index) + + def delete_field(self, field_name): + """ + 删除名为field_name的field - :param name: the name of the field to be deleted. + :param str field_name: 需要删除的field的名称. """ - self.field_arrays.pop(name) + self.field_arrays.pop(field_name) + + def has_field(self, field_name): + """ + 判断DataSet中是否有名为field_name这个field + :param str field_name: field的名称 + :return bool: 表示是否有名为field_name这个field + """ + if isinstance(field_name, str): + return field_name in self.field_arrays + return False + def get_field(self, field_name): + """ + 获取field_name这个field + + :param str field_name: field的名称 + :return: :class:`~fastNLP.FieldArray` + """ if field_name not in self.field_arrays: raise KeyError("Field name {} not found in DataSet".format(field_name)) return self.field_arrays[field_name] - + def get_all_fields(self): - """Return all the fields with their names. + """ + 返回一个dict，key为field_name, value为对应的 :class:`~fastNLP.FieldArray` - :return field_arrays: the internal data structure of DataSet. + :return: dict: 返回如上所述的字典 """ return self.field_arrays + + def get_field_names(self) -> list: + """ + 返回一个list，包含所有 field 的名字 + :return: list: 返回如上所述的列表 + """ + return sorted(self.field_arrays.keys()) + def get_length(self): - """Fetch the length of the dataset. + """ + 获取DataSet的元素数量 - :return length: + :return: int: DataSet中Instance的个数。 """ return len(self) - + def rename_field(self, old_name, new_name): - """Rename a field. + """ + 将某个field重新命名. - :param str old_name: - :param str new_name: + :param str old_name: 原来的field名称。 + :param str new_name: 修改为new_name。 """ if old_name in self.field_arrays: self.field_arrays[new_name] = self.field_arrays.pop(old_name) self.field_arrays[new_name].name = new_name else: raise KeyError("DataSet has no field named {}.".format(old_name)) - + def set_target(self, *field_names, flag=True): - """Change the target flag of these fields. + """ + 将field_names的field设置为target - :param field_names: a sequence of str, indicating field names - :param bool flag: Set these fields as target if True. Unset them if False. + Example:: + + dataset.set_target('labels', 'seq_len') # 将labels和seq_len这两个field的target属性设置为True + dataset.set_target('labels', 'seq_lens', flag=False) # 将labels和seq_len的target属性设置为False + + :param str field_names: field的名称 + :param bool flag: 将field_name的target状态设置为flag """ + assert isinstance(flag, bool), "Only bool type supported." for name in field_names: if name in self.field_arrays: self.field_arrays[name].is_target = flag else: raise KeyError("{} is not a valid field name.".format(name)) + + def set_input(self, *field_names, flag=True): + """ + 将field_names的field设置为input:: - def set_input(self, *field_name, flag=True): - """Set the input flag of these fields. + dataset.set_input('words', 'seq_len') # 将words和seq_len这两个field的input属性设置为True + dataset.set_input('words', flag=False) # 将words这个field的input属性设置为False - :param field_name: a sequence of str, indicating field names. - :param bool flag: Set these fields as input if True. Unset them if False. + :param str field_names: field的名称 + :param bool flag: 将field_name的input状态设置为flag """ - for name in field_name: + for name in field_names: if name in self.field_arrays: self.field_arrays[name].is_input = flag else: raise KeyError("{} is not a valid field name.".format(name)) + + def set_ignore_type(self, *field_names, flag=True): + """ + 将field设置为忽略类型状态。当某个field被设置了ignore_type, 则在被设置为target或者input时将不进行类型检查， + 默认情况下也不进行pad。 + + :param str field_names: field的名称 + :param bool flag: 将field_name的ignore_type状态设置为flag + :return: + """ + assert isinstance(flag, bool), "Only bool type supported." + for name in field_names: + if name in self.field_arrays: + self.field_arrays[name].ignore_type = flag + else: + raise KeyError("{} is not a valid field name.".format(name)) + + def set_padder(self, field_name, padder): + """ + 为field_name设置padder:: + + from fastNLP import EngChar2DPadder + padder = EngChar2DPadder() + dataset.set_padder('chars', padder) # 则chars这个field会使用EngChar2DPadder进行pad操作 + + :param str field_name: 设置field的padding方式为padder + :param None, Padder padder: 设置为None即删除padder, 即对该field不进行pad操作。 + """ + if field_name not in self.field_arrays: + raise KeyError("There is no field named {}.".format(field_name)) + self.field_arrays[field_name].set_padder(padder) + + def set_pad_val(self, field_name, pad_val): + """ + 为某个field设置对应的pad_val. + :param str field_name: 修改该field的pad_val + :param int pad_val: 该field的padder会以pad_val作为padding index + """ + if field_name not in self.field_arrays: + raise KeyError("There is no field named {}.".format(field_name)) + self.field_arrays[field_name].set_pad_val(pad_val) + def get_input_name(self): - """Get all field names with `is_input` as True. + """ + 返回所有is_input被设置为True的field名称 - :return field_names: a list of str + :return list: 里面的元素为被设置为input的field名称 """ return [name for name, field in self.field_arrays.items() if field.is_input] - + def get_target_name(self): - """Get all field names with `is_target` as True. + """ + 返回所有is_target被设置为True的field名称 - :return field_names: a list of str + :return list: 里面的元素为被设置为target的field名称 """ return [name for name, field in self.field_arrays.items() if field.is_target] + + def apply_field(self, func, field_name, new_field_name=None, **kwargs): + """ + 将DataSet中的每个instance中的名为 `field_name` 的field传给func，并获取它的返回值。 - def apply(self, func, new_field_name=None, **kwargs): - """Apply a function to every instance of the DataSet. + :param callable func: input是instance中名为 `field_name` 的field的内容。 + :param str field_name: 传入func的是哪个field。 + :param None,str new_field_name: 将func返回的内容放入到 `new_field_name` 这个field中，如果名称与已有的field相同，则覆 + 盖之前的field。如果为None则不创建新的field。 + :param optional kwargs: 支持输入is_input,is_target,ignore_type + + 1. is_input: bool, 如果为True则将名为 `new_field_name` 的field设置为input - :param func: a function that takes an instance as input. - :param str new_field_name: If not None, results of the function will be stored as a new field. - :param **kwargs: Accept parameters will be - (1) is_input: boolean, will be ignored if new_field is None. If True, the new field will be as input. - (2) is_target: boolean, will be ignored if new_field is None. If True, the new field will be as target. - :return results: if new_field_name is not passed, returned values of the function over all instances. + 2. is_target: bool, 如果为True则将名为 `new_field_name` 的field设置为target + + 3. ignore_type: bool, 如果为True则将名为 `new_field_name` 的field的ignore_type设置为true, 忽略其类型 + :return: list(Any), 里面的元素为func的返回值，所以list长度为DataSet的长度 + + """ + assert len(self) != 0, "Null DataSet cannot use apply_field()." + if field_name not in self: + raise KeyError("DataSet has no field named `{}`.".format(field_name)) + results = [] + idx = -1 + try: + for idx, ins in enumerate(self._inner_iter()): + results.append(func(ins[field_name])) + except Exception as e: + if idx != -1: + print("Exception happens at the `{}`th instance.".format(idx)) + raise e + if not (new_field_name is None) and len(list(filter(lambda x: x is not None, results))) == 0: # all None + raise ValueError("{} always return None.".format(_get_func_signature(func=func))) + + if new_field_name is not None: + self._add_apply_field(results, new_field_name, kwargs) + + return results + + def _add_apply_field(self, results, new_field_name, kwargs): """ - results = [func(ins) for ins in self._inner_iter()] - if len(list(filter(lambda x: x is not None, results))) == 0 and not (new_field_name is None): # all None - raise ValueError("{} always return None.".format(get_func_signature(func=func))) + 将results作为加入到新的field中，field名称为new_field_name + :param list(str) results: 一般是apply*()之后的结果 + :param str new_field_name: 新加入的field的名称 + :param dict kwargs: 用户apply*()时传入的自定义参数 + :return: + """ extra_param = {} if 'is_input' in kwargs: extra_param['is_input'] = kwargs['is_input'] if 'is_target' in kwargs: extra_param['is_target'] = kwargs['is_target'] - if new_field_name is not None: - if new_field_name in self.field_arrays: - # overwrite the field, keep same attributes - old_field = self.field_arrays[new_field_name] - if 'is_input' not in extra_param: - extra_param['is_input'] = old_field.is_input - if 'is_target' not in extra_param: - extra_param['is_target'] = old_field.is_target - self.add_field(name=new_field_name, - fields=results, - padding_val=old_field.padding_val, - **extra_param) - else: - self.add_field(name=new_field_name, fields=results, **extra_param) + if 'ignore_type' in kwargs: + extra_param['ignore_type'] = kwargs['ignore_type'] + if new_field_name in self.field_arrays: + # overwrite the field, keep same attributes + old_field = self.field_arrays[new_field_name] + if 'is_input' not in extra_param: + extra_param['is_input'] = old_field.is_input + if 'is_target' not in extra_param: + extra_param['is_target'] = old_field.is_target + if 'ignore_type' not in extra_param: + extra_param['ignore_type'] = old_field.ignore_type + self.add_field(field_name=new_field_name, fields=results, is_input=extra_param["is_input"], + is_target=extra_param["is_target"], ignore_type=extra_param['ignore_type']) else: - return results + self.add_field(field_name=new_field_name, fields=results, is_input=extra_param.get("is_input", None), + is_target=extra_param.get("is_target", None), + ignore_type=extra_param.get("ignore_type", False)) + + def apply(self, func, new_field_name=None, **kwargs): + """ + 将DataSet中每个instance传入到func中，并获取它的返回值. - def drop(self, func): - """Drop instances if a condition holds. + :param callable func: 参数是DataSet中的Instance + :param None,str new_field_name: 将func返回的内容放入到new_field_name这个field中，如果名称与已有的field相同，则覆 + 盖之前的field。如果为None则不创建新的field。 + :param optional kwargs: 支持输入is_input,is_target,ignore_type - :param func: a function that takes an Instance object as input, and returns bool. - The instance will be dropped if the function returns True. + 1. is_input: bool, 如果为True则将 `new_field_name` 的field设置为input + 2. is_target: bool, 如果为True则将 `new_field_name` 的field设置为target + + 3. ignore_type: bool, 如果为True则将 `new_field_name` 的field的ignore_type设置为true, 忽略其类型 + + :return: list(Any), 里面的元素为func的返回值，所以list长度为DataSet的长度 + """ + assert len(self) != 0, "Null DataSet cannot use apply()." + idx = -1 + try: + results = [] + for idx, ins in enumerate(self._inner_iter()): + results.append(func(ins)) + except Exception as e: + if idx != -1: + print("Exception happens at the `{}`th instance.".format(idx)) + raise e + # results = [func(ins) for ins in self._inner_iter()] + if not (new_field_name is None) and len(list(filter(lambda x: x is not None, results))) == 0: # all None + raise ValueError("{} always return None.".format(_get_func_signature(func=func))) + + if new_field_name is not None: + self._add_apply_field(results, new_field_name, kwargs) + + return results + + def drop(self, func, inplace=True): """ - results = [ins for ins in self._inner_iter() if not func(ins)] - for name, old_field in self.field_arrays.items(): - self.field_arrays[name].content = [ins[name] for ins in results] + func接受一个Instance，返回bool值。返回值为True时，该Instance会被移除或者加入到返回的DataSet中。 - def split(self, dev_ratio): - """Split the dataset into training and development(validation) set. + :param callable func: 接受一个Instance作为参数，返回bool值。为True时删除该instance + :param bool inplace: 是否在当前DataSet中直接删除instance。如果为False，被删除的Instance的组成的新DataSet将作为 + :返回值 + + :return: DataSet + """ + if inplace: + results = [ins for ins in self._inner_iter() if not func(ins)] + for name, old_field in self.field_arrays.items(): + self.field_arrays[name].content = [ins[name] for ins in results] + return self + else: + results = [ins for ins in self if not func(ins)] + if len(results) != 0: + dataset = DataSet(results) + for field_name, field in self.field_arrays.items(): + dataset.field_arrays[field_name].to(field) + return dataset + else: + return DataSet() + + def split(self, ratio): + """ + 将DataSet按照ratio的比例拆分，返回两个DataSet - :param float dev_ratio: the ratio of test set in all data. - :return (train_set, dev_set): - train_set: the training set - dev_set: the development set + :param float ratio: 0 1: + # list 跟 非list 混在一起 + raise RuntimeError("Mixed data types in Field {}: {}".format(self.name, list(type_set))) + # >1维list + inner_type_set = set() + for l in content: + [inner_type_set.add(type(obj)) for obj in l] + if list not in inner_type_set: + # 二维list + self.content_dim = 2 + return self._basic_type_detection(inner_type_set) + else: + if len(inner_type_set) == 1: + # >2维list + inner_inner_type_set = set() + for _2d_list in content: + for _1d_list in _2d_list: + [inner_inner_type_set.add(type(obj)) for obj in _1d_list] + if list in inner_inner_type_set: + raise RuntimeError("FieldArray cannot handle 4-D or more-D list.") + # 3维list + self.content_dim = 3 + return self._basic_type_detection(inner_inner_type_set) + else: + # list 跟 非list 混在一起 + raise RuntimeError("Mixed data types in Field {}: {}".format(self.name, list(inner_type_set))) + else: + # 一维list + for content_type in type_set: + if content_type not in self.BASIC_TYPES: + raise RuntimeError("Unexpected data type in Field '{}'. Expect one of {}. Got {}.".format( + self.name, self.BASIC_TYPES, content_type)) + self.content_dim = 1 + return self._basic_type_detection(type_set) + + def _basic_type_detection(self, type_set): + """ + :param type_set: a set of Python types + :return: one of self.BASIC_TYPES + """ + if len(type_set) == 1: + return type_set.pop() + elif len(type_set) == 2: + # 有多个basic type; 可能需要up-cast + if float in type_set and int in type_set: + # up-cast int to float + return float + else: + # str 跟 int 或者 float 混在一起 + raise RuntimeError("Mixed data types in Field {}: {}".format(self.name, list(type_set))) + else: + # str, int, float混在一起 + raise RuntimeError("Mixed data types in Field {}: {}".format(self.name, list(type_set))) + + def _1d_list_check(self, val): + """如果不是1D list就报错 + """ + type_set = set((type(obj) for obj in val)) + if any(obj not in self.BASIC_TYPES for obj in type_set): + raise ValueError("Mixed data types in Field {}: {}".format(self.name, list(type_set))) + self._basic_type_detection(type_set) + # otherwise: _basic_type_detection will raise error + return True + + def _2d_list_check(self, val): + """如果不是2D list 就报错 + """ + type_set = set(type(obj) for obj in val) + if list(type_set) != [list]: + raise ValueError("Mixed data types in Field {}: {}".format(self.name, type_set)) + inner_type_set = set() + for l in val: + for obj in l: + inner_type_set.add(type(obj)) + self._basic_type_detection(inner_type_set) + return True + + @staticmethod + def _map_to_np_type(basic_type): + type_mapping = {int: np.int64, float: np.float64, str: np.str, np.ndarray: np.ndarray} + return type_mapping[basic_type] + + def __repr__(self): + return "FieldArray {}: {}".format(self.name, self.content.__repr__()) + + def append(self, val): + """将val append到这个field的尾部。如果这个field已经被设置为input或者target，则在append之前会检查该类型是否与已有 + 的内容是匹配的。 + + :param Any val: 需要append的值。 + """ + if self.ignore_type is False: + if isinstance(val, list): + pass + elif isinstance(val, tuple): # 确保最外层是list + val = list(val) + elif isinstance(val, np.ndarray): + val = val.tolist() + elif any((isinstance(val, t) for t in self.BASIC_TYPES)): + pass + else: + raise RuntimeError( + "Unexpected data type {}. Should be list, np.array, or {}".format(type(val), self.BASIC_TYPES)) + + if self.is_input is True or self.is_target is True: + if type(val) == list: + if len(val) == 0: + raise ValueError("Cannot append an empty list.") + if self.content_dim == 2 and self._1d_list_check(val): + # 1维list检查 + pass + elif self.content_dim == 3 and self._2d_list_check(val): + # 2维list检查 + pass + else: + raise RuntimeError( + "Dimension not matched: expect dim={}, got {}.".format(self.content_dim - 1, val)) + elif type(val) in self.BASIC_TYPES and self.content_dim == 1: + # scalar检查 + if type(val) == float and self.pytype == int: + self.pytype = float + self.dtype = self._map_to_np_type(self.pytype) + else: + raise RuntimeError( + "Unexpected data type {}. Should be list, np.array, or {}".format(type(val), self.BASIC_TYPES)) + self.content.append(val) + + def __getitem__(self, indices): + return self.get(indices, pad=False) + + def __setitem__(self, idx, val): + assert isinstance(idx, int) + self.content[idx] = val + + def get(self, indices, pad=True): + """ + 根据给定的indices返回内容 + + :param int,List[int] indices: 获取indices对应的内容。 + :param bool pad: 是否对返回的结果进行padding。仅对indices为List[int]时有效 + :return: 根据给定的indices返回的内容，可能是单个值或List + """ + if isinstance(indices, int): + return self.content[indices] + if self.is_input is False and self.is_target is False: + raise RuntimeError("Please specify either is_input or is_target is True for {}".format(self.name)) + + contents = [self.content[i] for i in indices] + if self.padder is None or pad is False: + return np.array(contents) + else: + return self.padder(contents, field_name=self.name, field_ele_dtype=self.dtype) + + def set_padder(self, padder): + """ + 设置padder，在这个field进行pad的时候用这个padder进行pad，如果为None则不进行pad。 + + :param padder: :class:`~fastNLP.Padder` 类型，设置为None即删除padder。 + """ + if padder is not None: + assert isinstance(padder, Padder), "padder must be of type Padder." + self.padder = deepcopy(padder) + else: + self.padder = None + + def set_pad_val(self, pad_val): + """ + 修改padder的pad_val. + + :param int pad_val: 该field的pad值设置为该值。 + """ + if self.padder is not None: + self.padder.set_pad_val(pad_val) + return self + + def __len__(self): + """ + Returns the size of FieldArray. + + :return int length: + """ + return len(self.content) + + def to(self, other): + """ + 将other的属性复制给本FieldArray(other必须为FieldArray类型). + 属性包括 is_input, is_target, padder, ignore_type + + :param other: :class:`~fastNLP.FieldArray` 从哪个field拷贝属性 + :return: :class:`~fastNLP.FieldArray` + """ + assert isinstance(other, FieldArray), "Only support FieldArray type, not {}.".format(type(other)) + + self.is_input = other.is_input + self.is_target = other.is_target + self.padder = other.padder + self.ignore_type = other.ignore_type + + return self + + +def _is_iterable(content): + try: + _ = (e for e in content) + except TypeError: + return False + return True + + +class Padder: + """ + 别名：:class:`fastNLP.Padder` :class:`fastNLP.core.field.Padder` + + 所有padder都需要继承这个类，并覆盖__call__方法。 + 用于对batch进行padding操作。传入的element是inplace的，即直接修改element可能导致数据变化，建议inplace修改之前deepcopy一份。 + + .. py:function:: __call__(self, contents, field_name, field_ele_dtype): + 传入的是List内容。假设有以下的DataSet。 + + :param list(Any) contents: 传入的element是inplace的，即直接修改element可能导致数据变化，建议inplace修改之前 + deepcopy一份。 + :param str, field_name: field的名称。 + :param np.int64,np.float64,np.str,None, field_ele_dtype: 该field的内层元素的类型。如果该field的ignore_type为True，该这个值为None。 + :return: np.array([padded_element]) + + """ + + def __init__(self, pad_val=0, **kwargs): + self.pad_val = pad_val + + def set_pad_val(self, pad_val): + self.pad_val = pad_val + + def __call__(self, contents, field_name, field_ele_dtype): + """ + 传入的是List内容。假设有以下的DataSet。 + + :param list(Any) contents: 传入的element是inplace的，即直接修改element可能导致数据变化，建议inplace修改之前 + deepcopy一份。 + :param str, field_name: field的名称。 + :param np.int64,np.float64,np.str,None, field_ele_dtype: 该field的内层元素的类型。如果该field的ignore_type为True，该这个值为None。 + :return: np.array([padded_element]) + + Example:: + + from fastNLP import DataSet + from fastNLP import Instance + dataset = DataSet() + dataset.append(Instance(sent='this is a demo', length=4, + chars=[['t', 'h', 'i', 's'], ['i', 's'], ['a'], ['d', 'e', 'm', 'o']])) + dataset.append(Instance(sent='another one', length=2, + chars=[['a', 'n', 'o', 't', 'h', 'e', 'r'], ['o', 'n', 'e']])) + 如果调用 + batch = dataset.get([0,1], pad=True) + sent这个field的padder的__call__会接收到的内容会是 + [ + 'this is a demo', + 'another one' + ] + + length这个field的padder的__call__会接收到的内容会是 + [4, 2] + + chars这个field的padder的__call__会接收到的内容会是 + [ + [['t', 'h', 'i', 's'], ['i', 's'], ['a'], ['d', 'e', 'm', 'o']], + [['a', 'n', 'o', 't', 'h', 'e', 'r'], ['o', 'n', 'e']] + ] + + 即把每个instance中某个field的内容合成一个List传入 + + """ + raise NotImplementedError + + +class AutoPadder(Padder): + """ + 别名：:class:`fastNLP.AutoPadder` :class:`fastNLP.core.field.AutoPadder` + + 根据contents的数据自动判定是否需要做padding。 + + 1 如果元素类型(元素类型是指field中最里层元素的数据类型, 可以通过FieldArray.dtype查看，比如['This', 'is', ...]的元素类 + 型为np.str, [[1,2], ...]的元素类型为np.int64)的数据不为(np.int64, np.float64)则不会进行pad + + 2 如果元素类型为(np.int64, np.float64), + + 2.1 如果该field的内容为(np.int64, np.float64)，比如为seq_len, 则不进行padding + + 2.2 如果该field的内容为List, 那么会将Batch中的List pad为一样长。若该List下还有里层的List需要padding，请使用其它padder。 + 即如果Instance中field形如[1, 2, 3, ...]，则可以pad；若为[[1,2], [3,4, ...]]则不能进行pad + """ + + def __init__(self, pad_val=0): + """ + :param pad_val: int, padding的位置使用该index + """ + super().__init__(pad_val=pad_val) + + def _is_two_dimension(self, contents): + """ + 判断contents是不是只有两个维度。[[1,2], [3]]是两个维度. [[[1,2], [3, 4, 5]], [[4,5]]]有三个维度 + :param contents: + :return: + """ + value = contents[0] + if isinstance(value, (np.ndarray, list)): + value = value[0] + if isinstance(value, (np.ndarray, list)): + return False + return True + return False + + def __call__(self, contents, field_name, field_ele_dtype): + + if not _is_iterable(contents[0]): + array = np.array([content for content in contents], dtype=field_ele_dtype) + elif field_ele_dtype in (np.int64, np.float64) and self._is_two_dimension(contents): + max_len = max([len(content) for content in contents]) + array = np.full((len(contents), max_len), self.pad_val, dtype=field_ele_dtype) + for i, content in enumerate(contents): + array[i][:len(content)] = content + elif field_ele_dtype is None: + array = np.array(contents) # 当ignore_type=True时，直接返回contents + else: # should only be str + array = np.array([content for content in contents]) + return array + + +class EngChar2DPadder(Padder): + """ + 别名：:class:`fastNLP.EngChar2DPadder` :class:`fastNLP.core.field.EngChar2DPadder` + + 用于为英语执行character级别的2D padding操作。对应的field内容应该类似[['T', 'h', 'i', 's'], ['a'], ['d', 'e', 'm', 'o']]， + 但这个Padder只能处理index为int的情况。 + + padded过后的batch内容，形状为(batch_size, max_sentence_length, max_word_length). max_sentence_length为这个batch中最大句 + 子长度；max_word_length为这个batch中最长的word的长度:: + + from fastNLP import DataSet + from fastNLP import EngChar2DPadder + from fastNLP import Vocabulary + dataset = DataSet({'sent': ['This is the first demo', 'This is the second demo']}) + dataset.apply(lambda ins:[list(word) for word in ins['sent'].split()], new_field_name='chars') + vocab = Vocabulary() + vocab.from_dataset(dataset, field_name='chars') + vocab.index_dataset(dataset, field_name='chars') + dataset.set_input('chars') + padder = EngChar2DPadder() + dataset.set_padder('chars', padder) # chars这个field的设置为了EnChar2DPadder + + """ + + def __init__(self, pad_val=0, pad_length=0): + """ + :param pad_val: int, pad的位置使用该index + :param pad_length: int, 如果为0则取一个batch中最大的单词长度作为padding长度。如果为大于0的数，则将所有单词的长度 + 都pad或截取到该长度. + """ + super().__init__(pad_val=pad_val) + + self.pad_length = pad_length + + def _exactly_three_dims(self, contents, field_name): + """ + 检查传入的contents是否刚好是3维，如果不是3维就报错。理论上，第一个维度是batch，第二个维度是word，第三个维度是character + :param contents: + :param field_name: str + :return: + """ + if not isinstance(contents, list): + raise TypeError("contents should be a list, not {}.".format(type(contents))) + value = contents[0] + try: + value = value[0] + except: + raise ValueError("Field:{} only has one dimension.".format(field_name)) + try: + value = value[0] + except: + raise ValueError("Field:{} only has two dimensions.".format(field_name)) + + if _is_iterable(value): + raise ValueError("Field:{} has more than 3 dimension.".format(field_name)) + + def __call__(self, contents, field_name, field_ele_dtype): + """ + 期望输入类似于 + [ + [[0, 2], [2, 3, 4], ..], + [[9, 8, 2, 4], [1, 2,], ...], + .... + ] + + :param contents: + :param field_name: + :param field_ele_dtype + :return: + """ + if field_ele_dtype not in (np.int64, np.float64): + raise TypeError('dtype of Field:{} should be np.int64 or np.float64 to do 2D padding, get {}.'.format( + field_name, field_ele_dtype + )) + self._exactly_three_dims(contents, field_name) + if self.pad_length < 1: + max_char_length = max(max([[len(char_lst) for char_lst in word_lst] for word_lst in contents])) + else: + max_char_length = self.pad_length + max_sent_length = max(len(word_lst) for word_lst in contents) + batch_size = len(contents) + dtype = type(contents[0][0][0]) + + padded_array = np.full((batch_size, max_sent_length, max_char_length), fill_value=self.pad_val, + dtype=dtype) + for b_idx, word_lst in enumerate(contents): + for c_idx, char_lst in enumerate(word_lst): + chars = char_lst[:max_char_length] + padded_array[b_idx, c_idx, :len(chars)] = chars + + return padded_array diff --git a/fastNLP/core/fieldarray.py b/fastNLP/core/fieldarray.py deleted file mode 100644 index c1a2db1c..00000000 --- a/fastNLP/core/fieldarray.py +++ /dev/null @@ -1,188 +0,0 @@ -import numpy as np - - -class FieldArray(object): - """``FieldArray`` is the collection of ``Instance``s of the same field. - It is the basic element of ``DataSet`` class. - - :param str name: the name of the FieldArray - :param list content: a list of int, float, str or np.ndarray, or a list of list of one, or a np.ndarray. - :param int padding_val: the integer for padding. Default: 0. - :param bool is_target: If True, this FieldArray is used to compute loss. - :param bool is_input: If True, this FieldArray is used to the model input. - - """ - - def __init__(self, name, content, padding_val=0, is_target=None, is_input=None): - self.name = name - if isinstance(content, list): - content = content - elif isinstance(content, np.ndarray): - content = content.tolist() # convert np.ndarray into 2-D list - else: - raise TypeError("content in FieldArray can only be list or numpy.ndarray, got {}.".format(type(content))) - self.content = content - self.padding_val = padding_val - - self._is_target = None - self._is_input = None - - self.BASIC_TYPES = (int, float, str, np.ndarray) - self.is_2d_list = False - self.pytype = None # int, float, str, or np.ndarray - self.dtype = None # np.int64, np.float64, np.str - - if is_input is not None: - self.is_input = is_input - if is_target is not None: - self.is_target = is_target - - @property - def is_input(self): - return self._is_input - - @is_input.setter - def is_input(self, value): - if value is True: - self.pytype = self._type_detection(self.content) - self.dtype = self._map_to_np_type(self.pytype) - self._is_input = value - - @property - def is_target(self): - return self._is_target - - @is_target.setter - def is_target(self, value): - if value is True: - self.pytype = self._type_detection(self.content) - self.dtype = self._map_to_np_type(self.pytype) - self._is_target = value - - def _type_detection(self, content): - """ - - :param content: a list of int, float, str or np.ndarray, or a list of list of one. - :return type: one of int, float, str, np.ndarray - - """ - if isinstance(content, list) and len(content) > 0 and isinstance(content[0], list): - # content is a 2-D list - if not all(isinstance(_, list) for _ in content): # strict check 2-D list - raise TypeError("Please provide 2-D list.") - type_set = set([self._type_detection(x) for x in content]) - if len(type_set) == 2 and int in type_set and float in type_set: - type_set = {float} - elif len(type_set) > 1: - raise TypeError("Cannot create FieldArray with more than one type. Provided {}".format(type_set)) - self.is_2d_list = True - return type_set.pop() - - elif isinstance(content, list): - # content is a 1-D list - if len(content) == 0: - # the old error is not informative enough. - raise RuntimeError("Cannot create FieldArray with an empty list. Or one element in the list is empty.") - type_set = set([type(item) for item in content]) - - if len(type_set) == 1 and tuple(type_set)[0] in self.BASIC_TYPES: - return type_set.pop() - elif len(type_set) == 2 and float in type_set and int in type_set: - # up-cast int to float - return float - else: - raise TypeError("Cannot create FieldArray with type {}".format(*type_set)) - else: - raise TypeError("Cannot create FieldArray with type {}".format(type(content))) - - @staticmethod - def _map_to_np_type(basic_type): - type_mapping = {int: np.int64, float: np.float64, str: np.str, np.ndarray: np.ndarray} - return type_mapping[basic_type] - - def __repr__(self): - return "FieldArray {}: {}".format(self.name, self.content.__repr__()) - - def append(self, val): - """Add a new item to the tail of FieldArray. - - :param val: int, float, str, or a list of one. - """ - if self.is_target is True or self.is_input is True: - # only check type when used as target or input - - val_type = type(val) - if val_type == list: # shape check - if self.is_2d_list is False: - raise RuntimeError("Cannot append a list into a 1-D FieldArray. Please provide an element.") - if len(val) == 0: - raise RuntimeError("Cannot append an empty list.") - val_list_type = set([type(_) for _ in val]) # type check - if len(val_list_type) == 2 and int in val_list_type and float in val_list_type: - # up-cast int to float - val_type = float - elif len(val_list_type) == 1: - val_type = val_list_type.pop() - else: - raise TypeError("Cannot append a list of {}".format(val_list_type)) - else: - if self.is_2d_list is True: - raise RuntimeError("Cannot append a non-list into a 2-D list. Please provide a list.") - - if val_type == float and self.pytype == int: - # up-cast - self.pytype = float - self.dtype = self._map_to_np_type(self.pytype) - elif val_type == int and self.pytype == float: - pass - elif val_type == self.pytype: - pass - else: - raise TypeError("Cannot append type {} into type {}".format(val_type, self.pytype)) - - self.content.append(val) - - def __getitem__(self, indices): - return self.get(indices) - - def __setitem__(self, idx, val): - assert isinstance(idx, int) - self.content[idx] = val - - def get(self, indices): - """Fetch instances based on indices. - - :param indices: an int, or a list of int. - :return: - """ - if isinstance(indices, int): - return self.content[indices] - if self.is_input is False and self.is_target is False: - raise RuntimeError("Please specify either is_input or is_target is True for {}".format(self.name)) - batch_size = len(indices) - - if not is_iterable(self.content[0]): - array = np.array([self.content[i] for i in indices], dtype=self.dtype) - elif self.dtype in (np.int64, np.float64): - max_len = max([len(self.content[i]) for i in indices]) - array = np.full((batch_size, max_len), self.padding_val, dtype=self.dtype) - for i, idx in enumerate(indices): - array[i][:len(self.content[idx])] = self.content[idx] - else: # should only be str - array = np.array([self.content[i] for i in indices]) - return array - - def __len__(self): - """Returns the size of FieldArray. - - :return int length: - """ - return len(self.content) - - -def is_iterable(content): - try: - _ = (e for e in content) - except TypeError: - return False - return True diff --git a/fastNLP/core/instance.py b/fastNLP/core/instance.py index a102b51c..5408522e 100644 --- a/fastNLP/core/instance.py +++ b/fastNLP/core/instance.py @@ -1,36 +1,52 @@ -class Instance(object): - """An Instance is an example of data. - Example:: - ins = Instance(field_1=[1, 1, 1], field_2=[2, 2, 2]) - ins["field_1"] - >>[1, 1, 1] - ins.add_field("field_3", [3, 3, 3]) +""" +instance 模块实现了Instance 类在fastNLP中对应sample。一个sample可以认为是一个Instance类型的对象。 +便于理解的例子可以参考文档 :doc:`fastNLP.core.dataset` 中的表格 - :param fields: a dict of (str: list). +""" +__all__ = [ + "Instance" +] - """ +class Instance(object): + """ + 别名：:class:`fastNLP.Instance` :class:`fastNLP.core.instance.Instance` + + Instance是fastNLP中对应一个sample的类。每个sample在fastNLP中是一个Instance对象。 + Instance一般与 :class:`~fastNLP.DataSet` 一起使用, Instance的初始化如下面的Example所示:: + + >>>from fastNLP import Instance + >>>ins = Instance(field_1=[1, 1, 1], field_2=[2, 2, 2]) + >>>ins["field_1"] + [1, 1, 1] + >>>ins.add_field("field_3", [3, 3, 3]) + >>>ins = Instance(**{'x1': 1, 'x2':np.zeros((3, 4))}) + """ + def __init__(self, **fields): + self.fields = fields - + def add_field(self, field_name, field): - """Add a new field to the instance. + """ + 向Instance中增加一个field - :param field_name: str, the name of the field. + :param str field_name: 新增field的名称 + :param Any field: 新增field的内容 """ self.fields[field_name] = field - + def __getitem__(self, name): if name in self.fields: return self.fields[name] else: raise KeyError("{} not found".format(name)) - + def __setitem__(self, name, field): return self.add_field(name, field) - + def __repr__(self): s = '\'' return "{" + ",\n".join( - "\'" + field_name + "\': " + str(self.fields[field_name]) +\ + "\'" + field_name + "\': " + str(self.fields[field_name]) + \ f" type={(str(type(self.fields[field_name]))).split(s)[1]}" for field_name in self.fields) + "}" diff --git a/fastNLP/core/losses.py b/fastNLP/core/losses.py index 9b8b8d8f..9dc02f3d 100644 --- a/fastNLP/core/losses.py +++ b/fastNLP/core/losses.py @@ -1,33 +1,50 @@ +""" +losses 模块定义了 fastNLP 中所需的各种损失函数，一般做为 :class:`~fastNLP.Trainer` 的参数使用。 + +""" +__all__ = [ + "LossBase", + + "LossFunc", + "LossInForward", + + "CrossEntropyLoss", + "BCELoss", + "L1Loss", + "NLLLoss" +] + import inspect from collections import defaultdict import torch import torch.nn.functional as F -from fastNLP.core.utils import CheckError -from fastNLP.core.utils import CheckRes -from fastNLP.core.utils import _build_args -from fastNLP.core.utils import _check_arg_dict_list -from fastNLP.core.utils import _check_function_or_method -from fastNLP.core.utils import get_func_signature +from .utils import _CheckError +from .utils import _CheckRes +from .utils import _build_args +from .utils import _check_arg_dict_list +from .utils import _check_function_or_method +from .utils import _get_func_signature class LossBase(object): - """Base class for all losses. - """ + 所有loss的基类。如果想了解其中的原理，请查看源码。 + """ + def __init__(self): self.param_map = {} self._checked = False - + def get_loss(self, *args, **kwargs): raise NotImplementedError - + def _init_param_map(self, key_map=None, **kwargs): - """Check the validity of key_map and other param map. Add these into self.param_map + """检查key_map和其他参数map，并将这些映射关系添加到self.param_map - :param key_map: dict - :param kwargs: + :param dict key_map: 表示key的映射关系 + :param kwargs: key word args里面的每一个的键-值对都会被构造成映射关系 :return: None """ value_counter = defaultdict(set) @@ -55,21 +72,21 @@ class LossBase(object): for value, key_set in value_counter.items(): if len(key_set) > 1: raise ValueError(f"Several parameters:{key_set} are provided with one output {value}.") - + # check consistence between signature and param_map func_spect = inspect.getfullargspec(self.get_loss) func_args = [arg for arg in func_spect.args if arg != 'self'] for func_param, input_param in self.param_map.items(): if func_param not in func_args: raise NameError( - f"Parameter `{func_param}` is not in {get_func_signature(self.get_loss)}. Please check the " + f"Parameter `{func_param}` is not in {_get_func_signature(self.get_loss)}. Please check the " f"initialization parameters, or change its signature.") - + # evaluate should not have varargs. # if func_spect.varargs: # raise NameError(f"Delete `*{func_spect.varargs}` in {get_func_signature(self.get_loss)}(Do not use " # f"positional argument.).") - + def _fast_param_map(self, pred_dict, target_dict): """Only used as inner function. When the pred_dict, target is unequivocal. Don't need users to pass key_map. such as pred_dict has one element, target_dict has one element @@ -84,34 +101,34 @@ class LossBase(object): fast_param['target'] = list(target_dict.values())[0] return fast_param return fast_param - + def __call__(self, pred_dict, target_dict, check=False): """ - :param pred_dict: A dict from forward function of the network. - :param target_dict: A dict from DataSet.batch_y. - :param check: Boolean. Force to check the mapping functions when it is running. + :param dict pred_dict: 模型的forward函数返回的dict + :param dict target_dict: DataSet.batch_y里的键-值对所组成的dict + :param Boolean check: 每一次执行映射函数的时候是否检查映射表，默认为不检查 :return: """ fast_param = self._fast_param_map(pred_dict, target_dict) if fast_param: loss = self.get_loss(**fast_param) return loss - + if not self._checked: # 1. check consistence between signature and param_map func_spect = inspect.getfullargspec(self.get_loss) func_args = set([arg for arg in func_spect.args if arg != 'self']) for func_arg, input_arg in self.param_map.items(): if func_arg not in func_args: - raise NameError(f"`{func_arg}` not in {get_func_signature(self.get_loss)}.") - + raise NameError(f"`{func_arg}` not in {_get_func_signature(self.get_loss)}.") + # 2. only part of the param_map are passed, left are not for arg in func_args: if arg not in self.param_map: self.param_map[arg] = arg # This param does not need mapping. self._evaluate_args = func_args self._reverse_param_map = {input_arg: func_arg for func_arg, input_arg in self.param_map.items()} - + # need to wrap inputs in dict. mapped_pred_dict = {} mapped_target_dict = {} @@ -131,7 +148,7 @@ class LossBase(object): not_duplicate_flag += 1 if not_duplicate_flag == 3: duplicated.append(input_arg) - + # missing if not self._checked: check_res = _check_arg_dict_list(self.get_loss, [mapped_pred_dict, mapped_target_dict]) @@ -141,37 +158,50 @@ class LossBase(object): for idx, func_arg in enumerate(missing): # Don't delete `` in this information, nor add `` replaced_missing[idx] = f"{self.param_map[func_arg]}" + f"(assign to `{func_arg}` " \ - f"in `{self.__class__.__name__}`)" - - check_res = CheckRes(missing=replaced_missing, - unused=check_res.unused, - duplicated=duplicated, - required=check_res.required, - all_needed=check_res.all_needed, - varargs=check_res.varargs) - + f"in `{self.__class__.__name__}`)" + + check_res = _CheckRes(missing=replaced_missing, + unused=check_res.unused, + duplicated=duplicated, + required=check_res.required, + all_needed=check_res.all_needed, + varargs=check_res.varargs) + if check_res.missing or check_res.duplicated: - raise CheckError(check_res=check_res, - func_signature=get_func_signature(self.get_loss)) + raise _CheckError(check_res=check_res, + func_signature=_get_func_signature(self.get_loss)) refined_args = _build_args(self.get_loss, **mapped_pred_dict, **mapped_target_dict) - + loss = self.get_loss(**refined_args) self._checked = True - + return loss class LossFunc(LossBase): - """A wrapper of user-provided loss function. + """ + 别名：:class:`fastNLP.LossFunc` :class:`fastNLP.core.losses.LossFunc` + + 提供给用户使用自定义损失函数的类 + + :param func: 用户自行定义的损失函数，应当为一个函数或者callable(func)为True的ojbect + :param dict key_map: 参数映射表。键为Model/DataSet参数名，值为损失函数参数名。 + fastNLP的trainer将在训练时从模型返回值或者训练数据DataSet的target=True的field中 + 找到相对应的参数名为value的参数，并传入func中作为参数名为key的参数 + :param kwargs: 除了参数映射表以外可以用key word args的方式设置参数映射关系 + + 使用方法:: + + func = torch.nn.CrossEntropyLoss() + loss_func = LossFunc(func, input="pred", target="label") + # 这表示构建了一个损失函数类，由func计算损失函数，其中将从模型返回值或者DataSet的target=True的field + # 当中找到一个参数名为`pred`的参数传入func一个参数名为`input`的参数；找到一个参数名为`label`的参数 + # 传入func作为一个名为`target`的参数 """ + def __init__(self, func, key_map=None, **kwargs): - """ - - :param func: a callable object, such as a function. - :param dict key_map: - :param kwargs: - """ + super(LossFunc, self).__init__() _check_function_or_method(func) if key_map is not None: @@ -181,78 +211,129 @@ class LossFunc(LossBase): if len(kwargs) > 0: for key, val in kwargs.items(): self.param_map.update({key: val}) - + self.get_loss = func class CrossEntropyLoss(LossBase): + """ + 别名：:class:`fastNLP.CrossEntropyLoss` :class:`fastNLP.core.losses.CrossEntropyLoss` + + 交叉熵损失函数 + + :param pred: 参数映射表中 `pred` 的映射关系，None表示映射关系为 `pred` -> `pred` + :param target: 参数映射表中 `target` 的映射关系，None表示映射关系为 `target` -> `target` + :param padding_idx: padding的index，在计算loss时将忽略target中标号为padding_idx的内容 + + Example:: + + loss = CrossEntropyLoss(pred='pred', target='label', padding_idx=0) + + """ + def __init__(self, pred=None, target=None, padding_idx=-100): - # TODO 需要做一些检查，F.cross_entropy在计算时，如果pred是(16, 10 ,4), target的形状按道理应该是(16, 10), 但实际却需要 - # TODO （16， 4） + # TODO 需要做一些检查，F.cross_entropy在计算时，如果pred是(16, 10 ,4), target的形状按道理应该是(16, 10), 但实际需要（16，4） super(CrossEntropyLoss, self).__init__() self._init_param_map(pred=pred, target=target) self.padding_idx = padding_idx - + def get_loss(self, pred, target): return F.cross_entropy(input=pred, target=target, ignore_index=self.padding_idx) class L1Loss(LossBase): + """ + 别名：:class:`fastNLP.L1Loss` :class:`fastNLP.core.losses.L1Loss` + + L1损失函数 + + :param pred: 参数映射表中 `pred` 的映射关系，None表示映射关系为 `pred` -> `pred` + :param target: 参数映射表中 `target` 的映射关系，None表示映射关系为 `target` >`target` + + """ + def __init__(self, pred=None, target=None): super(L1Loss, self).__init__() self._init_param_map(pred=pred, target=target) - + def get_loss(self, pred, target): return F.l1_loss(input=pred, target=target) class BCELoss(LossBase): + """ + 别名：:class:`fastNLP.BCELoss` :class:`fastNLP.core.losses.BCELoss` + + 二分类交叉熵损失函数 + + :param pred: 参数映射表中`pred`的映射关系，None表示映射关系为`pred`->`pred` + :param target: 参数映射表中`target`的映射关系，None表示映射关系为`target`->`target` + """ + def __init__(self, pred=None, target=None): super(BCELoss, self).__init__() self._init_param_map(pred=pred, target=target) - + def get_loss(self, pred, target): return F.binary_cross_entropy(input=pred, target=target) class NLLLoss(LossBase): + """ + 别名：:class:`fastNLP.NLLLoss` :class:`fastNLP.core.losses.NLLLoss` + + 负对数似然损失函数 + + :param pred: 参数映射表中`pred`的映射关系，None表示映射关系为`pred`->`pred` + :param target: 参数映射表中`target`的映射关系，None表示映射关系为`target`->`target` + """ + def __init__(self, pred=None, target=None): super(NLLLoss, self).__init__() self._init_param_map(pred=pred, target=target) - + def get_loss(self, pred, target): return F.nll_loss(input=pred, target=target) class LossInForward(LossBase): + """ + 别名：:class:`fastNLP.LossInForward` :class:`fastNLP.core.losses.LossInForward` + + 从forward()函数返回结果中获取loss + + :param str loss_key: 在forward函数中loss的键名，默认为loss + """ + def __init__(self, loss_key='loss'): super().__init__() if not isinstance(loss_key, str): raise TypeError(f"Only str allowed for loss_key, got {type(loss_key)}.") self.loss_key = loss_key - + def get_loss(self, **kwargs): if self.loss_key not in kwargs: - check_res = CheckRes(missing=[self.loss_key + f"(assign to `{self.loss_key}` " \ - f"in `{self.__class__.__name__}`"], - unused=[], - duplicated=[], - required=[], - all_needed=[], - varargs=[]) - raise CheckError(check_res=check_res, func_signature=get_func_signature(self.get_loss)) + check_res = _CheckRes( + missing=[self.loss_key + f"(assign to `{self.loss_key}` in `{self.__class__.__name__}`"], + unused=[], + duplicated=[], + required=[], + all_needed=[], + varargs=[]) + raise _CheckError(check_res=check_res, func_signature=_get_func_signature(self.get_loss)) return kwargs[self.loss_key] - + def __call__(self, pred_dict, target_dict, check=False): - + loss = self.get_loss(**pred_dict) - + if not (isinstance(loss, torch.Tensor) and len(loss.size()) == 0): if not isinstance(loss, torch.Tensor): raise TypeError(f"Loss excepted to be a torch.Tensor, got {type(loss)}") - raise RuntimeError(f"The size of loss excepts to be torch.Size([]), got {loss.size()}") - + loss = torch.sum(loss) / (loss.view(-1)).size(0) + # raise RuntimeError(f"The size of loss excepts to be torch.Size([]), got {loss.size()}") + return loss @@ -271,7 +352,7 @@ def squash(predict, truth, **kwargs): :param predict: Tensor, model output :param truth: Tensor, truth from dataset - :param **kwargs: extra arguments + :param kwargs: extra arguments :return predict , truth: predict & truth after processing """ return predict.view(-1, predict.size()[-1]), truth.view(-1, ) @@ -315,20 +396,20 @@ def mask(predict, truth, **kwargs): :param predict: Tensor, [batch_size , max_len , tag_size] :param truth: Tensor, [batch_size , max_len] - :param **kwargs: extra arguments, kwargs["mask"]: ByteTensor, [batch_size , max_len], the mask Tensor. The position that is 1 will be selected. + :param kwargs: extra arguments, kwargs["mask"]: ByteTensor, [batch_size , max_len], the mask Tensor. The position that is 1 will be selected. :return predict , truth: predict & truth after processing """ if kwargs.get("mask") is None: return predict, truth mask = kwargs["mask"] - + predict, truth = squash(predict, truth) mask = mask.view(-1, ) - + predict = torch.masked_select(predict.permute(1, 0), mask).view(predict.size()[-1], -1).permute(1, 0) truth = torch.masked_select(truth, mask) - + return predict, truth @@ -343,4 +424,3 @@ def make_mask(lens, tar_len): mask = [torch.ge(lens, i + 1) for i in range(tar_len)] mask = torch.stack(mask, 1) return mask - diff --git a/fastNLP/core/metrics.py b/fastNLP/core/metrics.py index dfb20480..f633a80f 100644 --- a/fastNLP/core/metrics.py +++ b/fastNLP/core/metrics.py @@ -1,49 +1,134 @@ +""" +metrics 模块实现了 fastNLP 所需的各种常用衡量指标，一般做为 :class:`~fastNLP.Trainer` 的参数使用。 + +""" +__all__ = [ + "MetricBase", + "AccuracyMetric", + "SpanFPreRecMetric", + "SQuADMetric" +] + import inspect from collections import defaultdict import numpy as np import torch -from fastNLP.core.utils import CheckError -from fastNLP.core.utils import CheckRes -from fastNLP.core.utils import _build_args -from fastNLP.core.utils import _check_arg_dict_list -from fastNLP.core.utils import get_func_signature -from fastNLP.core.utils import seq_lens_to_masks -from fastNLP.core.vocabulary import Vocabulary +from .utils import _CheckError +from .utils import _CheckRes +from .utils import _build_args +from .utils import _check_arg_dict_list +from .utils import _get_func_signature +from .utils import seq_len_to_mask +from .vocabulary import Vocabulary class MetricBase(object): - """Base class for all metrics. - - ``MetricBase`` handles validity check of its input dictionaries - ``pred_dict`` and ``target_dict``. - ``pred_dict`` is the output of ``forward()`` or prediction function of a model. - ``target_dict`` is the ground truth from DataSet where ``is_target`` is set ``True``. - ``MetricBase`` will do the following type checks: - - 1. whether self.evaluate has varargs, which is not supported. - 2. whether params needed by self.evaluate is not included in ``pred_dict``, ``target_dict``. - 3. whether params needed by self.evaluate duplicate in ``pred_dict``, ``target_dict``. - 4. whether params in ``pred_dict``, ``target_dict`` are not used by evaluate.(Might cause warning) - - Besides, before passing params into self.evaluate, this function will filter out params from output_dict and - target_dict which are not used in self.evaluate. (but if **kwargs presented in self.evaluate, no filtering - will be conducted.) - However, in some cases where type check is not necessary, ``_fast_param_map`` will be used. + """ + 所有metrics的基类,，所有的传入到Trainer, Tester的Metric需要继承自该对象，需要覆盖写入evaluate(), get_metric()方法。 + + evaluate(xxx)中传入的是一个batch的数据。 + + get_metric(xxx)当所有数据处理完毕，调用该方法得到最终的metric值 + + 以分类问题中，Accuracy计算为例 + 假设model的forward返回dict中包含'pred'这个key, 并且该key需要用于Accuracy:: + + class Model(nn.Module): + def __init__(xxx): + # do something + def forward(self, xxx): + # do something + return {'pred': pred, 'other_keys':xxx} # pred's shape: batch_size x num_classes + + 假设dataset中'label'这个field是需要预测的值，并且该field被设置为了target + 对应的AccMetric可以按如下的定义, version1, 只使用这一次:: + + class AccMetric(MetricBase): + def __init__(self): + super().__init__() + + # 根据你的情况自定义指标 + self.corr_num = 0 + self.total = 0 + + def evaluate(self, label, pred): # 这里的名称需要和dataset中target field与model返回的key是一样的，不然找不到对应的value + # dev或test时，每个batch结束会调用一次该方法，需要实现如何根据每个batch累加metric + self.total += label.size(0) + self.corr_num += label.eq(pred).sum().item() + + def get_metric(self, reset=True): # 在这里定义如何计算metric + acc = self.corr_num/self.total + if reset: # 是否清零以便重新计算 + self.corr_num = 0 + self.total = 0 + return {'acc': acc} # 需要返回一个dict，key为该metric的名称，该名称会显示到Trainer的progress bar中 + + + version2，如果需要复用Metric，比如下一次使用AccMetric时，dataset中目标field不叫label而叫y，或者model的输出不是pred:: + + class AccMetric(MetricBase): + def __init__(self, label=None, pred=None): + # 假设在另一场景使用时，目标field叫y，model给出的key为pred_y。则只需要在初始化AccMetric时， + # acc_metric = AccMetric(label='y', pred='pred_y')即可。 + # 当初始化为acc_metric = AccMetric()，即label=None, pred=None, fastNLP会直接使用'label', 'pred'作为key去索取对 + # 应的的值 + super().__init__() + self._init_param_map(label=label, pred=pred) # 该方法会注册label和pred. 仅需要注册evaluate()方法会用到的参数名即可 + # 如果没有注册该则效果与version1就是一样的 + + # 根据你的情况自定义指标 + self.corr_num = 0 + self.total = 0 + + def evaluate(self, label, pred): # 这里的参数名称需要和self._init_param_map()注册时一致。 + # dev或test时，每个batch结束会调用一次该方法，需要实现如何根据每个batch累加metric + self.total += label.size(0) + self.corr_num += label.eq(pred).sum().item() + + def get_metric(self, reset=True): # 在这里定义如何计算metric + acc = self.corr_num/self.total + if reset: # 是否清零以便重新计算 + self.corr_num = 0 + self.total = 0 + return {'acc': acc} # 需要返回一个dict，key为该metric的名称，该名称会显示到Trainer的progress bar中 + + + ``MetricBase`` 将会在输入的字典 ``pred_dict`` 和 ``target_dict`` 中进行检查. + ``pred_dict`` 是模型当中 ``forward()`` 函数或者 ``predict()`` 函数的返回值. + ``target_dict`` 是DataSet当中的ground truth, 判定ground truth的条件是field的 ``is_target`` 被设置为True. + + ``MetricBase`` 会进行以下的类型检测: + + 1. self.evaluate当中是否有varargs, 这是不支持的. + 2. self.evaluate当中所需要的参数是否既不在 ``pred_dict`` 也不在 ``target_dict`` . + 3. self.evaluate当中所需要的参数是否既在 ``pred_dict`` 也在 ``target_dict`` . + + 除此以外，在参数被传入self.evaluate以前，这个函数会检测 ``pred_dict`` 和 ``target_dict`` 当中没有被用到的参数 + 如果kwargs是self.evaluate的参数，则不会检测 + + + self.evaluate将计算一个批次(batch)的评价指标，并累计。 没有返回值 + self.get_metric将统计当前的评价指标并返回评价结果, 返回值需要是一个dict, key是指标名称，value是指标的值 """ + def __init__(self): self.param_map = {} # key is param in function, value is input param. self._checked = False - + def evaluate(self, *args, **kwargs): raise NotImplementedError - + + def get_metric(self, reset=True): + raise NotImplemented + def _init_param_map(self, key_map=None, **kwargs): - """Check the validity of key_map and other param map. Add these into self.param_map + """检查key_map和其他参数map，并将这些映射关系添加到self.param_map - :param key_map: dict - :param kwargs: + :param dict key_map: 表示key的映射关系 + :param kwargs: key word args里面的每一个的键-值对都会被构造成映射关系 :return: None """ value_counter = defaultdict(set) @@ -71,19 +156,16 @@ class MetricBase(object): for value, key_set in value_counter.items(): if len(key_set) > 1: raise ValueError(f"Several parameters:{key_set} are provided with one output {value}.") - + # check consistence between signature and param_map func_spect = inspect.getfullargspec(self.evaluate) func_args = [arg for arg in func_spect.args if arg != 'self'] for func_param, input_param in self.param_map.items(): if func_param not in func_args: raise NameError( - f"Parameter `{func_param}` is not in {get_func_signature(self.evaluate)}. Please check the " + f"Parameter `{func_param}` is not in {_get_func_signature(self.evaluate)}. Please check the " f"initialization parameters, or change its signature.") - - def get_metric(self, reset=True): - raise NotImplemented - + def _fast_param_map(self, pred_dict, target_dict): """Only used as inner function. When the pred_dict, target is unequivocal. Don't need users to pass key_map. such as pred_dict has one element, target_dict has one element @@ -95,49 +177,47 @@ class MetricBase(object): fast_param = {} if len(self.param_map) == 2 and len(pred_dict) == 1 and len(target_dict) == 1: fast_param['pred'] = list(pred_dict.values())[0] - fast_param['target'] = list(pred_dict.values())[0] + fast_param['target'] = list(target_dict.values())[0] return fast_param return fast_param - + def __call__(self, pred_dict, target_dict): """ - - This method will call self.evaluate method. - Before calling self.evaluate, it will first check the validity of output_dict, target_dict - (1) whether params needed by self.evaluate is not included in output_dict,target_dict. - (2) whether params needed by self.evaluate duplicate in pred_dict, target_dict - (3) whether params in output_dict, target_dict are not used by evaluate.(Might cause warning) - Besides, before passing params into self.evaluate, this function will filter out params from output_dict and - target_dict which are not used in self.evaluate. (but if **kwargs presented in self.evaluate, no filtering - will be conducted.) - This function also support _fast_param_map. - :param pred_dict: usually the output of forward or prediction function - :param target_dict: usually features set as target.. + 这个方法会调用self.evaluate 方法. + 在调用之前，会进行以下检测: + 1. self.evaluate当中是否有varargs, 这是不支持的. + 2. self.evaluate当中所需要的参数是否既不在``pred_dict``也不在``target_dict``. + 3. self.evaluate当中所需要的参数是否既在``pred_dict``也在``target_dict``. + + 除此以外，在参数被传入self.evaluate以前，这个函数会检测``pred_dict``和``target_dict``当中没有被用到的参数 + 如果kwargs是self.evaluate的参数，则不会检测 + :param pred_dict: 模型的forward函数或者predict函数返回的dict + :param target_dict: DataSet.batch_y里的键-值对所组成的dict(即is_target=True的fields的内容) :return: """ - if not callable(self.evaluate): - raise TypeError(f"{self.__class__.__name__}.evaluate has to be callable, not {type(self.evaluate)}.") - - fast_param = self._fast_param_map(pred_dict=pred_dict, target_dict=target_dict) + + fast_param = self._fast_param_map(pred_dict, target_dict) if fast_param: self.evaluate(**fast_param) return - + if not self._checked: + if not callable(self.evaluate): + raise TypeError(f"{self.__class__.__name__}.evaluate has to be callable, not {type(self.evaluate)}.") # 1. check consistence between signature and param_map func_spect = inspect.getfullargspec(self.evaluate) func_args = set([arg for arg in func_spect.args if arg != 'self']) for func_arg, input_arg in self.param_map.items(): if func_arg not in func_args: - raise NameError(f"`{func_arg}` not in {get_func_signature(self.evaluate)}.") - + raise NameError(f"`{func_arg}` not in {_get_func_signature(self.evaluate)}.") + # 2. only part of the param_map are passed, left are not for arg in func_args: if arg not in self.param_map: self.param_map[arg] = arg # This param does not need mapping. self._evaluate_args = func_args self._reverse_param_map = {input_arg: func_arg for func_arg, input_arg in self.param_map.items()} - + # need to wrap inputs in dict. mapped_pred_dict = {} mapped_target_dict = {} @@ -157,7 +237,7 @@ class MetricBase(object): not_duplicate_flag += 1 if not_duplicate_flag == 3: duplicated.append(input_arg) - + # missing if not self._checked: check_res = _check_arg_dict_list(self.evaluate, [mapped_pred_dict, mapped_target_dict]) @@ -168,141 +248,118 @@ class MetricBase(object): for idx, func_arg in enumerate(missing): # Don't delete `` in this information, nor add `` replaced_missing[idx] = f"{self.param_map[func_arg]}" + f"(assign to `{func_arg}` " \ - f"in `{self.__class__.__name__}`)" - - check_res = CheckRes(missing=replaced_missing, - unused=check_res.unused, - duplicated=duplicated, - required=check_res.required, - all_needed=check_res.all_needed, - varargs=check_res.varargs) - + f"in `{self.__class__.__name__}`)" + + check_res = _CheckRes(missing=replaced_missing, + unused=check_res.unused, + duplicated=duplicated, + required=check_res.required, + all_needed=check_res.all_needed, + varargs=check_res.varargs) + if check_res.missing or check_res.duplicated: - raise CheckError(check_res=check_res, - func_signature=get_func_signature(self.evaluate)) + raise _CheckError(check_res=check_res, + func_signature=_get_func_signature(self.evaluate)) refined_args = _build_args(self.evaluate, **mapped_pred_dict, **mapped_target_dict) - + self.evaluate(**refined_args) self._checked = True - + return class AccuracyMetric(MetricBase): - """Accuracy Metric - """ - def __init__(self, pred=None, target=None, seq_lens=None): + + 别名：:class:`fastNLP.AccuracyMetric` :class:`fastNLP.core.metrics.AccuracyMetric` + + 准确率Metric（其它的Metric参见 :doc:`fastNLP.core.metrics` ） + + :param pred: 参数映射表中 `pred` 的映射关系，None表示映射关系为 `pred` -> `pred` + :param target: 参数映射表中 `target` 的映射关系，None表示映射关系为 `target` -> `target` + :param seq_len: 参数映射表中 `seq_len` 的映射关系，None表示映射关系为 `seq_len` -> `seq_len` + """ + + def __init__(self, pred=None, target=None, seq_len=None): + super().__init__() - - self._init_param_map(pred=pred, target=target, seq_lens=seq_lens) - + + self._init_param_map(pred=pred, target=target, seq_len=seq_len) + self.total = 0 self.acc_count = 0 - - def _fast_param_map(self, pred_dict, target_dict): - """Only used as inner function. When the pred_dict, target is unequivocal. Don't need users to pass key_map. - such as pred_dict has one element, target_dict has one element - - :param pred_dict: - :param target_dict: - :return: dict, if dict is not None, pass it to self.evaluate. Otherwise do mapping. - """ - fast_param = {} - targets = list(target_dict.values()) - if len(targets) == 1 and isinstance(targets[0], torch.Tensor): - if len(pred_dict) == 1: - pred = list(pred_dict.values())[0] - fast_param['pred'] = pred - elif len(pred_dict) == 2: - pred1 = list(pred_dict.values())[0] - pred2 = list(pred_dict.values())[1] - if not (isinstance(pred1, torch.Tensor) and isinstance(pred2, torch.Tensor)): - return fast_param - if len(pred1.size()) < len(pred2.size()) and len(pred1.size()) == 1: - seq_lens = pred1 - pred = pred2 - elif len(pred1.size()) > len(pred2.size()) and len(pred2.size()) == 1: - seq_lens = pred2 - pred = pred1 - else: - return fast_param - fast_param['pred'] = pred - fast_param['seq_lens'] = seq_lens - else: - return fast_param - fast_param['target'] = targets[0] - # TODO need to make sure they all have same batch_size - return fast_param - - def evaluate(self, pred, target, seq_lens=None): + + def evaluate(self, pred, target, seq_len=None): """ + evaluate函数将针对一个批次的预测结果做评价指标的累计 - :param pred: List of (torch.Tensor, or numpy.ndarray). Element's shape can be: - torch.Size([B,]), torch.Size([B, n_classes]), torch.Size([B, max_len]), torch.Size([B, max_len, n_classes]) - :param target: List of (torch.Tensor, or numpy.ndarray). Element's can be: - torch.Size([B,]), torch.Size([B,]), torch.Size([B, max_len]), torch.Size([B, max_len]) - :param seq_lens: List of (torch.Tensor, or numpy.ndarray). Element's can be: - None, None, torch.Size([B], torch.Size([B]). ignored if masks are provided. + :param torch.Tensor pred: 预测的tensor, tensor的形状可以是torch.Size([B,]), torch.Size([B, n_classes]), + torch.Size([B, max_len]), 或者torch.Size([B, max_len, n_classes]) + :param torch.Tensor target: 真实值的tensor, tensor的形状可以是Element's can be: torch.Size([B,]), + torch.Size([B,]), torch.Size([B, max_len]), 或者torch.Size([B, max_len]) + :param torch.Tensor seq_len: 序列长度标记, 标记的形状可以是None, None, torch.Size([B]), 或者torch.Size([B]). + 如果mask也被传进来的话seq_len会被忽略. """ # TODO 这里报错需要更改，因为pred是啥用户并不知道。需要告知用户真实的value if not isinstance(pred, torch.Tensor): - raise TypeError(f"`pred` in {get_func_signature(self.evaluate)} must be torch.Tensor," + raise TypeError(f"`pred` in {_get_func_signature(self.evaluate)} must be torch.Tensor," f"got {type(pred)}.") if not isinstance(target, torch.Tensor): - raise TypeError(f"`target` in {get_func_signature(self.evaluate)} must be torch.Tensor," + raise TypeError(f"`target` in {_get_func_signature(self.evaluate)} must be torch.Tensor," f"got {type(target)}.") - - if seq_lens is not None and not isinstance(seq_lens, torch.Tensor): - raise TypeError(f"`seq_lens` in {get_func_signature(self.evaluate)} must be torch.Tensor," - f"got {type(seq_lens)}.") - - if seq_lens is not None: - masks = seq_lens_to_masks(seq_lens=seq_lens, float=True) + + if seq_len is not None and not isinstance(seq_len, torch.Tensor): + raise TypeError(f"`seq_lens` in {_get_func_signature(self.evaluate)} must be torch.Tensor," + f"got {type(seq_len)}.") + + if seq_len is not None: + masks = seq_len_to_mask(seq_len=seq_len) else: masks = None - + if pred.size() == target.size(): pass elif len(pred.size()) == len(target.size()) + 1: pred = pred.argmax(dim=-1) else: - raise RuntimeError(f"In {get_func_signature(self.evaluate)}, when pred have " + raise RuntimeError(f"In {_get_func_signature(self.evaluate)}, when pred have " f"size:{pred.size()}, target should have size: {pred.size()} or " f"{pred.size()[:-1]}, got {target.size()}.") - - pred = pred.float() - target = target.float() - + + target = target.to(pred) if masks is not None: - self.acc_count += torch.sum(torch.eq(pred, target).float() * masks.float()).item() - self.total += torch.sum(masks.float()).item() + self.acc_count += torch.sum(torch.eq(pred, target).masked_fill(masks.eq(0), 0)).item() + self.total += torch.sum(masks).item() else: - self.acc_count += torch.sum(torch.eq(pred, target).float()).item() + self.acc_count += torch.sum(torch.eq(pred, target)).item() self.total += np.prod(list(pred.size())) - + def get_metric(self, reset=True): - """Returns computed metric. + """ + get_metric函数将根据evaluate函数累计的评价指标统计量来计算最终的评价结果. - :param bool reset: whether to recount next time. - :return evaluate_result: {"acc": float} + :param bool reset: 在调用完get_metric后是否清空评价指标统计量. + :return dict evaluate_result: {"acc": float} """ - evaluate_result = {'acc': round(self.acc_count / self.total, 6)} + evaluate_result = {'acc': round(float(self.acc_count) / (self.total + 1e-12), 6)} if reset: self.acc_count = 0 self.total = 0 return evaluate_result -def bmes_tag_to_spans(tags, ignore_labels=None): + +def _bmes_tag_to_spans(tags, ignore_labels=None): """ + 给定一个tags的lis，比如['S-song', 'B-singer', 'M-singer', 'E-singer', 'S-moive', 'S-actor']。 + 返回[('song', (0, 1)), ('singer', (1, 4)), ('moive', (4, 5)), ('actor', (5, 6))] (左闭右开区间) :param tags: List[str], :param ignore_labels: List[str], 在该list中的label将被忽略 :return: List[Tuple[str, List[int, int]]]. [(label，[start, end])] """ ignore_labels = set(ignore_labels) if ignore_labels else set() - + spans = [] prev_bmes_tag = None for idx, tag in enumerate(tags): @@ -310,25 +367,59 @@ def bmes_tag_to_spans(tags, ignore_labels=None): bmes_tag, label = tag[:1], tag[2:] if bmes_tag in ('b', 's'): spans.append((label, [idx, idx])) - elif bmes_tag in ('m', 'e') and prev_bmes_tag in ('b', 'm') and label==spans[-1][0]: + elif bmes_tag in ('m', 'e') and prev_bmes_tag in ('b', 'm') and label == spans[-1][0]: spans[-1][1][1] = idx else: spans.append((label, [idx, idx])) prev_bmes_tag = bmes_tag - return [(span[0], (span[1][0], span[1][1])) - for span in spans - if span[0] not in ignore_labels + return [(span[0], (span[1][0], span[1][1] + 1)) + for span in spans + if span[0] not in ignore_labels ] -def bio_tag_to_spans(tags, ignore_labels=None): + +def _bmeso_tag_to_spans(tags, ignore_labels=None): """ + 给定一个tags的lis，比如['O', 'B-singer', 'M-singer', 'E-singer', 'O', 'O']。 + 返回[('singer', (1, 4))] (左闭右开区间) :param tags: List[str], :param ignore_labels: List[str], 在该list中的label将被忽略 :return: List[Tuple[str, List[int, int]]]. [(label，[start, end])] """ ignore_labels = set(ignore_labels) if ignore_labels else set() + + spans = [] + prev_bmes_tag = None + for idx, tag in enumerate(tags): + tag = tag.lower() + bmes_tag, label = tag[:1], tag[2:] + if bmes_tag in ('b', 's'): + spans.append((label, [idx, idx])) + elif bmes_tag in ('m', 'e') and prev_bmes_tag in ('b', 'm') and label == spans[-1][0]: + spans[-1][1][1] = idx + elif bmes_tag == 'o': + pass + else: + spans.append((label, [idx, idx])) + prev_bmes_tag = bmes_tag + return [(span[0], (span[1][0], span[1][1] + 1)) + for span in spans + if span[0] not in ignore_labels + ] + +def _bio_tag_to_spans(tags, ignore_labels=None): + """ + 给定一个tags的lis，比如['O', 'B-singer', 'I-singer', 'I-singer', 'O', 'O']。 + 返回[('singer', (1, 4))] (左闭右开区间) + + :param tags: List[str], + :param ignore_labels: List[str], 在该list中的label将被忽略 + :return: List[Tuple[str, List[int, int]]]. [(label，[start, end])] + """ + ignore_labels = set(ignore_labels) if ignore_labels else set() + spans = [] prev_bio_tag = None for idx, tag in enumerate(tags): @@ -336,130 +427,139 @@ def bio_tag_to_spans(tags, ignore_labels=None): bio_tag, label = tag[:1], tag[2:] if bio_tag == 'b': spans.append((label, [idx, idx])) - elif bio_tag == 'i' and prev_bio_tag in ('b', 'i') and label==spans[-1][0]: + elif bio_tag == 'i' and prev_bio_tag in ('b', 'i') and label == spans[-1][0]: spans[-1][1][1] = idx - elif bio_tag == 'o': # o tag does not count + elif bio_tag == 'o': # o tag does not count pass else: spans.append((label, [idx, idx])) prev_bio_tag = bio_tag - return [(span[0], (span[1][0], span[1][1])) - for span in spans - if span[0] not in ignore_labels - ] + return [(span[0], (span[1][0], span[1][1] + 1)) for span in spans if span[0] not in ignore_labels] class SpanFPreRecMetric(MetricBase): """ + 别名：:class:`fastNLP.SpanFPreRecMetric` :class:`fastNLP.core.metrics.SpanFPreRecMetric` + 在序列标注问题中，以span的方式计算F, pre, rec. - 最后得到的metric结果为 - { - 'f': xxx, # 这里使用f考虑以后可以计算f_beta值 - 'pre': xxx, - 'rec':xxx - } - 若only_gross=False, 即还会返回各个label的metric统计值 + 比如中文Part of speech中，会以character的方式进行标注，句子'中国在亚洲'对应的POS可能为(以BMES为例) + ['B-NN', 'E-NN', 'S-DET', 'B-NN', 'E-NN']。该metric就是为类似情况下的F1计算。 + 最后得到的metric结果为:: + { - 'f': xxx, - 'pre': xxx, - 'rec':xxx, - 'f-label': xxx, - 'pre-label': xxx, - 'rec-label':xxx, - ... - } + 'f': xxx, # 这里使用f考虑以后可以计算f_beta值 + 'pre': xxx, + 'rec':xxx + } + + 若only_gross=False, 即还会返回各个label的metric统计值:: + + { + 'f': xxx, + 'pre': xxx, + 'rec':xxx, + 'f-label': xxx, + 'pre-label': xxx, + 'rec-label':xxx, + ... + } + :param tag_vocab: 标签的 :class:`~fastNLP.Vocabulary` 。支持的标签为"B"(没有label)；或"B-xxx"(xxx为某种label，比如POS中的NN)， + 在解码时，会将相同xxx的认为是同一个label，比如['B-NN', 'E-NN']会被合并为一个'NN'. + :param str pred: 用该key在evaluate()时从传入dict中取出prediction数据。 为None，则使用'pred'取数据 + :param str target: 用该key在evaluate()时从传入dict中取出target数据。 为None，则使用'target'取数据 + :param str seq_len: 用该key在evaluate()时从传入dict中取出sequence length数据。为None，则使用'seq_len'取数据。 + :param str encoding_type: 目前支持bio, bmes + :param list ignore_labels: str 组成的list. 这个list中的class不会被用于计算。例如在POS tagging时传入['NN']，则不会计算'NN'这 + 个label + :param bool only_gross: 是否只计算总的f1, precision, recall的值；如果为False，不仅返回总的f1, pre, rec, 还会返回每个 + label的f1, pre, rec + :param str f_type: 'micro'或'macro'. 'micro':通过先计算总体的TP，FN和FP的数量，再计算f, precision, recall; 'macro': + 分布计算每个类别的f, precision, recall，然后做平均（各类别f的权重相同） + :param float beta: f_beta分数，f_beta = (1 + beta^2)*(pre*rec)/(beta^2*pre + rec). 常用为beta=0.5, 1, 2. 若为0.5 + 则精确率的权重高于召回率；若为1，则两者平等；若为2，则召回率权重高于精确率。 """ - def __init__(self, tag_vocab, pred=None, target=None, seq_lens=None, encoding_type='bio', ignore_labels=None, - only_gross=True, f_type='micro', beta=1): - """ - - :param tag_vocab: Vocabulary, 标签的vocabulary。支持的标签为"B"(没有label)；或"B-xxx"(xxx为某种label，比如POS中的NN)， - 在解码时，会将相同xxx的认为是同一个label，比如['B-NN', 'E-NN']会被合并为一个'NN'. - :param pred: str, 用该key在evaluate()时从传入dict中取出prediction数据。 为None，则使用'pred'取数据 - :param target: str, 用该key在evaluate()时从传入dict中取出target数据。 为None，则使用'target'取数据 - :param seq_lens: str, 用该key在evaluate()时从传入dict中取出sequence length数据。为None，则使用'seq_lens'取数据。 - :param encoding_type: str, 目前支持bio, bmes - :param ignore_labels, List[str]. 这个list中的class不会被用于计算。例如在POS tagging时传入['NN']，则不会计算'NN'这 - 个label - :param only_gross, bool. 是否只计算总的f1, precision, recall的值；如果为False，不仅返回总的f1, pre, rec, 还会返回每个 - label的f1, pre, rec - :param f_type, str. 'micro'或'macro'. 'micro':通过先计算总体的TP，FN和FP的数量，再计算f, precision, recall; 'macro': - 分布计算每个类别的f, precision, recall，然后做平均（各类别f的权重相同） - :param beta, float. f_beta分数，f_beta = (1 + beta^2)*(pre*rec)/(beta^2*pre + rec). 常用为beta=0.5, 1, 2. 若为0.5 - 则精确率的权重高于召回率；若为1，则两者平等；若为2，则召回率权重高于精确率。 - """ + + def __init__(self, tag_vocab, pred=None, target=None, seq_len=None, encoding_type='bio', ignore_labels=None, + only_gross=True, f_type='micro', beta=1): + encoding_type = encoding_type.lower() - if encoding_type not in ('bio', 'bmes'): - raise ValueError("Only support 'bio' or 'bmes' type.") + if not isinstance(tag_vocab, Vocabulary): raise TypeError("tag_vocab can only be fastNLP.Vocabulary, not {}.".format(type(tag_vocab))) if f_type not in ('micro', 'macro'): raise ValueError("f_type only supports `micro` or `macro`', got {}.".format(f_type)) - + self.encoding_type = encoding_type if self.encoding_type == 'bmes': - self.tag_to_span_func = bmes_tag_to_spans + self.tag_to_span_func = _bmes_tag_to_spans elif self.encoding_type == 'bio': - self.tag_to_span_func = bio_tag_to_spans + self.tag_to_span_func = _bio_tag_to_spans + elif self.encoding_type == 'bmeso': + self.tag_to_span_func = _bmeso_tag_to_spans + else: + raise ValueError("Only support 'bio', 'bmes', 'bmeso' type.") + self.ignore_labels = ignore_labels self.f_type = f_type self.beta = beta - self.beta_square = self.beta**2 + self.beta_square = self.beta ** 2 self.only_gross = only_gross - + super().__init__() - self._init_param_map(pred=pred, target=target, seq_lens=seq_lens) - + self._init_param_map(pred=pred, target=target, seq_len=seq_len) + self.tag_vocab = tag_vocab - + self._true_positives = defaultdict(int) self._false_positives = defaultdict(int) self._false_negatives = defaultdict(int) + + def evaluate(self, pred, target, seq_len): + """evaluate函数将针对一个批次的预测结果做评价指标的累计 - def evaluate(self, pred, target, seq_lens): - """ - A lot of design idea comes from allennlp's measure - :param pred: - :param target: - :param seq_lens: + :param pred: [batch, seq_len] 或者 [batch, seq_len, len(tag_vocab)], 预测的结果 + :param target: [batch, seq_len], 真实值 + :param seq_len: [batch] 文本长度标记 :return: """ if not isinstance(pred, torch.Tensor): - raise TypeError(f"`pred` in {get_func_signature(self.evaluate)} must be torch.Tensor," + raise TypeError(f"`pred` in {_get_func_signature(self.evaluate)} must be torch.Tensor," f"got {type(pred)}.") if not isinstance(target, torch.Tensor): - raise TypeError(f"`target` in {get_func_signature(self.evaluate)} must be torch.Tensor," + raise TypeError(f"`target` in {_get_func_signature(self.evaluate)} must be torch.Tensor," f"got {type(target)}.") - - if not isinstance(seq_lens, torch.Tensor): - raise TypeError(f"`seq_lens` in {get_func_signature(self.evaluate)} must be torch.Tensor," - f"got {type(seq_lens)}.") - + + if not isinstance(seq_len, torch.Tensor): + raise TypeError(f"`seq_lens` in {_get_func_signature(self.evaluate)} must be torch.Tensor," + f"got {type(seq_len)}.") + if pred.size() == target.size() and len(target.size()) == 2: pass elif len(pred.size()) == len(target.size()) + 1 and len(target.size()) == 2: - pred = pred.argmax(dim=-1) num_classes = pred.size(-1) + pred = pred.argmax(dim=-1) if (target >= num_classes).any(): raise ValueError("A gold label passed to SpanBasedF1Metric contains an " "id >= {}, the number of classes.".format(num_classes)) else: - raise RuntimeError(f"In {get_func_signature(self.evaluate)}, when pred have " + raise RuntimeError(f"In {_get_func_signature(self.evaluate)}, when pred have " f"size:{pred.size()}, target should have size: {pred.size()} or " f"{pred.size()[:-1]}, got {target.size()}.") - + batch_size = pred.size(0) + pred = pred.tolist() + target = target.tolist() for i in range(batch_size): - pred_tags = pred[i, :int(seq_lens[i])].tolist() - gold_tags = target[i, :int(seq_lens[i])].tolist() - + pred_tags = pred[i][:int(seq_len[i])] + gold_tags = target[i][:int(seq_len[i])] + pred_str_tags = [self.tag_vocab.to_word(tag) for tag in pred_tags] gold_str_tags = [self.tag_vocab.to_word(tag) for tag in gold_tags] - + pred_spans = self.tag_to_span_func(pred_str_tags, ignore_labels=self.ignore_labels) gold_spans = self.tag_to_span_func(gold_str_tags, ignore_labels=self.ignore_labels) - + for span in pred_spans: if span in gold_spans: self._true_positives[span[0]] += 1 @@ -468,10 +568,11 @@ class SpanFPreRecMetric(MetricBase): self._false_positives[span[0]] += 1 for span in gold_spans: self._false_negatives[span[0]] += 1 - + def get_metric(self, reset=True): + """get_metric函数将根据evaluate函数累计的评价指标统计量来计算最终的评价结果.""" evaluate_result = {} - if not self.only_gross or self.f_type=='macro': + if not self.only_gross or self.f_type == 'macro': tags = set(self._false_negatives.keys()) tags.update(set(self._false_positives.keys())) tags.update(set(self._true_positives.keys())) @@ -486,34 +587,37 @@ class SpanFPreRecMetric(MetricBase): f_sum += f pre_sum += pre rec_sum + rec - if not self.only_gross and tag!='': # tag!=''防止无tag的情况 + if not self.only_gross and tag != '': # tag!=''防止无tag的情况 f_key = 'f-{}'.format(tag) pre_key = 'pre-{}'.format(tag) rec_key = 'rec-{}'.format(tag) evaluate_result[f_key] = f evaluate_result[pre_key] = pre evaluate_result[rec_key] = rec - + if self.f_type == 'macro': - evaluate_result['f'] = f_sum/len(tags) - evaluate_result['pre'] = pre_sum/len(tags) - evaluate_result['rec'] = rec_sum/len(tags) - + evaluate_result['f'] = f_sum / len(tags) + evaluate_result['pre'] = pre_sum / len(tags) + evaluate_result['rec'] = rec_sum / len(tags) + if self.f_type == 'micro': f, pre, rec = self._compute_f_pre_rec(sum(self._true_positives.values()), sum(self._false_negatives.values()), sum(self._false_positives.values())) - evaluate_result['f'] = round(f, 6) - evaluate_result['pre'] = round(pre, 6) - evaluate_result['rec'] = round(rec, 6) - + evaluate_result['f'] = f + evaluate_result['pre'] = pre + evaluate_result['rec'] = rec + if reset: self._true_positives = defaultdict(int) self._false_positives = defaultdict(int) self._false_negatives = defaultdict(int) - + + for key, value in evaluate_result.items(): + evaluate_result[key] = round(value, 6) + return evaluate_result - + def _compute_f_pre_rec(self, tp, fn, fp): """ @@ -525,137 +629,9 @@ class SpanFPreRecMetric(MetricBase): pre = tp / (fp + tp + 1e-13) rec = tp / (fn + tp + 1e-13) f = (1 + self.beta_square) * pre * rec / (self.beta_square * pre + rec + 1e-13) - + return f, pre, rec -class BMESF1PreRecMetric(MetricBase): - """ - 按照BMES标注方式计算f1, precision, recall。由于可能存在非法tag，比如"BS"，所以需要用以下的表格做转换，cur_B意思是当前tag是B， - next_B意思是后一个tag是B。则cur_B=S，即将当前被predict是B的tag标为S；next_M=B, 即将后一个被predict是M的tag标为B - | | next_B | next_M | next_E | next_S | end | - |:-----:|:-------:|:--------:|:--------:|:-------:|:-------:| - | start | 合法 | next_M=B | next_E=S | 合法 | - | - | cur_B | cur_B=S | 合法 | 合法 | cur_B=S | cur_B=S | - | cur_M | cur_M=E | 合法 | 合法 | cur_M=E | cur_M=E | - | cur_E | 合法 | next_M=B | next_E=S | 合法 | 合法 | - | cur_S | 合法 | next_M=B | next_E=S | 合法 | 合法 | - 举例： - prediction为BSEMS，会被认为是SSSSS. - - 本Metric不检验target的合法性，请务必保证target的合法性。 - pred的形状应该为(batch_size, max_len) 或 (batch_size, max_len, 4)。 - target形状为 (batch_size, max_len) - seq_lens形状为 (batch_size, ) - - """ - - def __init__(self, b_idx=0, m_idx=1, e_idx=2, s_idx=3, pred=None, target=None, seq_lens=None): - """ - 需要申明BMES这四种tag中，各种tag对应的idx。所有不为b_idx, m_idx, e_idx, s_idx的数字都认为是s_idx。 - - :param b_idx: int, Begin标签所对应的tag idx. - :param m_idx: int, Middle标签所对应的tag idx. - :param e_idx: int, End标签所对应的tag idx. - :param s_idx: int, Single标签所对应的tag idx - :param pred: str, 用该key在evaluate()时从传入dict中取出prediction数据。 为None，则使用'pred'取数据 - :param target: str, 用该key在evaluate()时从传入dict中取出target数据。 为None，则使用'target'取数据 - :param seq_lens: str, 用该key在evaluate()时从传入dict中取出seqence length数据。为None，则使用'seq_lens'取数据。 - """ - super().__init__() - - self._init_param_map(pred=pred, target=target, seq_lens=seq_lens) - - self.yt_wordnum = 0 - self.yp_wordnum = 0 - self.corr_num = 0 - - self.b_idx = b_idx - self.m_idx = m_idx - self.e_idx = e_idx - self.s_idx = s_idx - # 还原init处介绍的矩阵 - self._valida_matrix = { - -1: [(-1, -1), (1, self.b_idx), (1, self.s_idx), (-1, -1)], # magic start idx - self.b_idx:[(0, self.s_idx), (-1, -1), (-1, -1), (0, self.s_idx), (0, self.s_idx)], - self.m_idx:[(0, self.e_idx), (-1, -1), (-1, -1), (0, self.e_idx), (0, self.e_idx)], - self.e_idx:[(-1, -1), (1, self.b_idx), (1, self.s_idx), (-1, -1), (-1, -1)], - self.s_idx:[(-1, -1), (1, self.b_idx), (1, self.s_idx), (-1, -1), (-1, -1)], - } - - def _validate_tags(self, tags): - """ - 给定一个tag的Tensor，返回合法tag - - :param tags: Tensor, shape: (seq_len, ) - :return: 返回修改为合法tag的list - """ - assert len(tags)!=0 - assert isinstance(tags, torch.Tensor) and len(tags.size())==1 - padded_tags = [-1, *tags.tolist(), -1] - for idx in range(len(padded_tags)-1): - cur_tag = padded_tags[idx] - if cur_tag not in self._valida_matrix: - cur_tag = self.s_idx - if padded_tags[idx+1] not in self._valida_matrix: - padded_tags[idx+1] = self.s_idx - next_tag = padded_tags[idx+1] - shift_tag = self._valida_matrix[cur_tag][next_tag] - if shift_tag[0]!=-1: - padded_tags[idx+shift_tag[0]] = shift_tag[1] - - return padded_tags[1:-1] - - def evaluate(self, pred, target, seq_lens): - if not isinstance(pred, torch.Tensor): - raise TypeError(f"`pred` in {get_func_signature(self.evaluate)} must be torch.Tensor," - f"got {type(pred)}.") - if not isinstance(target, torch.Tensor): - raise TypeError(f"`target` in {get_func_signature(self.evaluate)} must be torch.Tensor," - f"got {type(target)}.") - - if not isinstance(seq_lens, torch.Tensor): - raise TypeError(f"`seq_lens` in {get_func_signature(self.evaluate)} must be torch.Tensor," - f"got {type(seq_lens)}.") - - if pred.size() == target.size() and len(target.size()) == 2: - pass - elif len(pred.size()) == len(target.size()) + 1 and len(target.size()) == 2: - pred = pred.argmax(dim=-1) - else: - raise RuntimeError(f"In {get_func_signature(self.evaluate)}, when pred have " - f"size:{pred.size()}, target should have size: {pred.size()} or " - f"{pred.size()[:-1]}, got {target.size()}.") - - for idx in range(len(pred)): - seq_len = seq_lens[idx] - target_tags = target[idx][:seq_len].tolist() - pred_tags = pred[idx][:seq_len] - pred_tags = self._validate_tags(pred_tags) - start_idx = 0 - for t_idx, (t_tag, p_tag) in enumerate(zip(target_tags, pred_tags)): - if t_tag in (self.s_idx, self.e_idx): - self.yt_wordnum += 1 - corr_flag = True - for i in range(start_idx, t_idx+1): - if target_tags[i]!=pred_tags[i]: - corr_flag = False - if corr_flag: - self.corr_num += 1 - start_idx = t_idx + 1 - if p_tag in (self.s_idx, self.e_idx): - self.yp_wordnum += 1 - - def get_metric(self, reset=True): - P = self.corr_num / (self.yp_wordnum + 1e-12) - R = self.corr_num / (self.yt_wordnum + 1e-12) - F = 2 * P * R / (P + R + 1e-12) - evaluate_result = {'f': round(F, 6), 'pre':round(P, 6), 'rec': round(R, 6)} - if reset: - self.yp_wordnum = 0 - self.yt_wordnum = 0 - self.corr_num = 0 - return evaluate_result - def _prepare_metrics(metrics): """ @@ -688,7 +664,7 @@ def _prepare_metrics(metrics): return _metrics -def accuracy_topk(y_true, y_prob, k=1): +def _accuracy_topk(y_true, y_prob, k=1): """Compute accuracy of y_true matching top-k probable labels in y_prob. :param y_true: ndarray, true label, [n_samples] @@ -704,7 +680,7 @@ def accuracy_topk(y_true, y_prob, k=1): return acc -def pred_topk(y_prob, k=1): +def _pred_topk(y_prob, k=1): """Return top-k predicted labels and corresponding probabilities. :param y_prob: ndarray, size [n_samples, n_classes], probabilities on labels @@ -720,3 +696,171 @@ def pred_topk(y_prob, k=1): (1, k)) y_prob_topk = y_prob[x_axis_index, y_pred_topk] return y_pred_topk, y_prob_topk + + +class SQuADMetric(MetricBase): + """ + 别名：:class:`fastNLP.SQuADMetric` :class:`fastNLP.core.metrics.SQuADMetric` + + SQuAD数据集metric + + :param pred1: 参数映射表中`pred1`的映射关系，None表示映射关系为`pred1`->`pred1` + :param pred2: 参数映射表中`pred2`的映射关系，None表示映射关系为`pred2`->`pred2` + :param target1: 参数映射表中`target1`的映射关系，None表示映射关系为`target1`->`target1` + :param target2: 参数映射表中`target2`的映射关系，None表示映射关系为`target2`->`target2` + :param float beta: f_beta分数，f_beta = (1 + beta^2)*(pre*rec)/(beta^2*pre + rec). 常用为beta=0.5, 1, 2. 若为0.5 + 则精确率的权重高于召回率；若为1，则两者平等；若为2，则召回率权重高于精确率。 + :param bool right_open: right_open为true表示start跟end指针指向一个左闭右开区间，为false表示指向一个左闭右闭区间。 + :param bool print_predict_stat: True则输出预测答案是否为空与正确答案是否为空的统计信息, False则不输出 + + """ + + def __init__(self, pred1=None, pred2=None, target1=None, target2=None, + beta=1, right_open=True, print_predict_stat=False): + + super(SQuADMetric, self).__init__() + + self._init_param_map(pred1=pred1, pred2=pred2, target1=target1, target2=target2) + + self.print_predict_stat = print_predict_stat + + self.no_ans_correct = 0 + self.no_ans_wrong = 0 + + self.has_ans_correct = 0 + self.has_ans_wrong = 0 + + self.has_ans_f = 0. + + self.no2no = 0 + self.no2yes = 0 + self.yes2no = 0 + self.yes2yes = 0 + + self.f_beta = beta + + self.right_open = right_open + + def evaluate(self, pred1, pred2, target1, target2): + """evaluate函数将针对一个批次的预测结果做评价指标的累计 + + :param pred1: [batch]或者[batch, seq_len], 预测答案开始的index, 如果SQuAD2.0中答案为空则为0 + :param pred2: [batch]或者[batch, seq_len] 预测答案结束的index, 如果SQuAD2.0中答案为空则为0(左闭右闭区间)或者1(左闭右开区间) + :param target1: [batch], 正确答案开始的index, 如果SQuAD2.0中答案为空则为0 + :param target2: [batch], 正确答案结束的index, 如果SQuAD2.0中答案为空则为0(左闭右闭区间)或者1(左闭右开区间) + :return: None + """ + pred_start = pred1 + pred_end = pred2 + target_start = target1 + target_end = target2 + + if len(pred_start.size()) == 2: + start_inference = pred_start.max(dim=-1)[1].cpu().tolist() + else: + start_inference = pred_start.cpu().tolist() + if len(pred_end.size()) == 2: + end_inference = pred_end.max(dim=-1)[1].cpu().tolist() + else: + end_inference = pred_end.cpu().tolist() + + start, end = [], [] + max_len = pred_start.size(1) + t_start = target_start.cpu().tolist() + t_end = target_end.cpu().tolist() + + for s, e in zip(start_inference, end_inference): + start.append(min(s, e)) + end.append(max(s, e)) + for s, e, ts, te in zip(start, end, t_start, t_end): + if not self.right_open: + e += 1 + te += 1 + if ts == 0 and te == int(not self.right_open): + if s == 0 and e == int(not self.right_open): + self.no_ans_correct += 1 + self.no2no += 1 + else: + self.no_ans_wrong += 1 + self.no2yes += 1 + else: + if s == 0 and e == int(not self.right_open): + self.yes2no += 1 + else: + self.yes2yes += 1 + + if s == ts and e == te: + self.has_ans_correct += 1 + else: + self.has_ans_wrong += 1 + a = [0] * s + [1] * (e - s) + [0] * (max_len - e) + b = [0] * ts + [1] * (te - ts) + [0] * (max_len - te) + a, b = torch.tensor(a), torch.tensor(b) + + TP = int(torch.sum(a * b)) + pre = TP / int(torch.sum(a)) if int(torch.sum(a)) > 0 else 0 + rec = TP / int(torch.sum(b)) if int(torch.sum(b)) > 0 else 0 + + if pre + rec > 0: + f = (1 + (self.f_beta ** 2)) * pre * rec / ((self.f_beta ** 2) * pre + rec) + else: + f = 0 + self.has_ans_f += f + + def get_metric(self, reset=True): + """get_metric函数将根据evaluate函数累计的评价指标统计量来计算最终的评价结果.""" + evaluate_result = {} + + if self.no_ans_correct + self.no_ans_wrong + self.has_ans_correct + self.no_ans_wrong <= 0: + return evaluate_result + + evaluate_result['EM'] = 0 + evaluate_result[f'f_{self.f_beta}'] = 0 + + flag = 0 + + if self.no_ans_correct + self.no_ans_wrong > 0: + evaluate_result[f'noAns-f_{self.f_beta}'] = \ + round(100 * self.no_ans_correct / (self.no_ans_correct + self.no_ans_wrong), 3) + evaluate_result['noAns-EM'] = \ + round(100 * self.no_ans_correct / (self.no_ans_correct + self.no_ans_wrong), 3) + evaluate_result[f'f_{self.f_beta}'] += evaluate_result[f'noAns-f_{self.f_beta}'] + evaluate_result['EM'] += evaluate_result['noAns-EM'] + flag += 1 + + if self.has_ans_correct + self.has_ans_wrong > 0: + evaluate_result[f'hasAns-f_{self.f_beta}'] = \ + round(100 * self.has_ans_f / (self.has_ans_correct + self.has_ans_wrong), 3) + evaluate_result['hasAns-EM'] = \ + round(100 * self.has_ans_correct / (self.has_ans_correct + self.has_ans_wrong), 3) + evaluate_result[f'f_{self.f_beta}'] += evaluate_result[f'hasAns-f_{self.f_beta}'] + evaluate_result['EM'] += evaluate_result['hasAns-EM'] + flag += 1 + + if self.print_predict_stat: + evaluate_result['no2no'] = self.no2no + evaluate_result['no2yes'] = self.no2yes + evaluate_result['yes2no'] = self.yes2no + evaluate_result['yes2yes'] = self.yes2yes + + if flag <= 0: + return evaluate_result + + evaluate_result[f'f_{self.f_beta}'] = round(evaluate_result[f'f_{self.f_beta}'] / flag, 3) + evaluate_result['EM'] = round(evaluate_result['EM'] / flag, 3) + + if reset: + self.no_ans_correct = 0 + self.no_ans_wrong = 0 + + self.has_ans_correct = 0 + self.has_ans_wrong = 0 + + self.has_ans_f = 0. + + self.no2no = 0 + self.no2yes = 0 + self.yes2no = 0 + self.yes2yes = 0 + + return evaluate_result diff --git a/fastNLP/core/optimizer.py b/fastNLP/core/optimizer.py index 145f117c..ef619042 100644 --- a/fastNLP/core/optimizer.py +++ b/fastNLP/core/optimizer.py @@ -1,57 +1,82 @@ +""" +optimizer 模块定义了 fastNLP 中所需的各种优化器，一般做为 :class:`~fastNLP.Trainer` 的参数使用。 + +""" +__all__ = [ + "Optimizer", + "SGD", + "Adam" +] + import torch class Optimizer(object): """ + 别名：:class:`fastNLP.Optimizer` :class:`fastNLP.core.optimizer.Optimizer` - :param model_params: a generator. E.g. ``model.parameters()`` for PyTorch models. - :param kwargs: additional parameters. + :param model_params: a generator. E.g. ``model.parameters()`` for PyTorch models. + :param kwargs: additional parameters. """ + def __init__(self, model_params, **kwargs): if model_params is not None and not hasattr(model_params, "__next__"): raise RuntimeError("model parameters should be a generator, rather than {}.".format(type(model_params))) self.model_params = model_params self.settings = kwargs + + def construct_from_pytorch(self, model_params): + raise NotImplementedError + + def _get_require_grads_param(self, params): + """ + 将params中不需要gradient的删除 + :param iterable params: parameters + :return: list(nn.Parameters) + """ + return [param for param in params if param.requires_grad] class SGD(Optimizer): """ + 别名：:class:`fastNLP.SGD` :class:`fastNLP.core.optimizer.SGD` - :param float lr: learning rate. Default: 0.01 - :param float momentum: momentum. Default: 0 - :param model_params: a generator. E.g. ``model.parameters()`` for PyTorch models. + :param float lr: learning rate. Default: 0.01 + :param float momentum: momentum. Default: 0 + :param model_params: a generator. E.g. ``model.parameters()`` for PyTorch models. """ - + def __init__(self, lr=0.001, momentum=0, model_params=None): if not isinstance(lr, float): raise TypeError("learning rate has to be float.") super(SGD, self).__init__(model_params, lr=lr, momentum=momentum) - + def construct_from_pytorch(self, model_params): if self.model_params is None: # careful! generator cannot be assigned. - return torch.optim.SGD(model_params, **self.settings) + return torch.optim.SGD(self._get_require_grads_param(model_params), **self.settings) else: - return torch.optim.SGD(self.model_params, **self.settings) + return torch.optim.SGD(self._get_require_grads_param(self.model_params), **self.settings) class Adam(Optimizer): """ + 别名：:class:`fastNLP.Adam` :class:`fastNLP.core.optimizer.Adam` - :param float lr: learning rate - :param float weight_decay: - :param model_params: a generator. E.g. ``model.parameters()`` for PyTorch models. + :param float lr: learning rate + :param float weight_decay: + :param model_params: a generator. E.g. ``model.parameters()`` for PyTorch models. """ - + def __init__(self, lr=0.001, weight_decay=0, betas=(0.9, 0.999), eps=1e-8, amsgrad=False, model_params=None): if not isinstance(lr, float): raise TypeError("learning rate has to be float.") super(Adam, self).__init__(model_params, lr=lr, betas=betas, eps=eps, amsgrad=amsgrad, weight_decay=weight_decay) - + def construct_from_pytorch(self, model_params): if self.model_params is None: # careful! generator cannot be assigned. - return torch.optim.Adam(model_params, **self.settings) + return torch.optim.Adam(self._get_require_grads_param(model_params), **self.settings) else: - return torch.optim.Adam(self.model_params, **self.settings) + return torch.optim.Adam(self._get_require_grads_param(self.model_params), **self.settings) diff --git a/fastNLP/core/predictor.py b/fastNLP/core/predictor.py index de9ddc8c..4f37e105 100644 --- a/fastNLP/core/predictor.py +++ b/fastNLP/core/predictor.py @@ -1,11 +1,20 @@ +""" + ..todo:: + 检查这个类是否需要 +""" +from collections import defaultdict + import torch -from fastNLP.core.batch import Batch -from fastNLP.core.sampler import SequentialSampler +from . import Batch +from . import DataSet +from . import SequentialSampler +from .utils import _build_args class Predictor(object): - """An interface for predicting outputs based on trained models. + """ + An interface for predicting outputs based on trained models. It does not care about evaluations of the model, which is different from Tester. This is a high-level model wrapper to be called by FastNLP. @@ -13,37 +22,55 @@ class Predictor(object): Currently, Predictor does not support GPU. """ - def __init__(self): + def __init__(self, network): + if not isinstance(network, torch.nn.Module): + raise ValueError( + "Only fastNLP.models.BaseModel or torch.nn,Module is allowed, not {}".format(type(network))) + self.network = network self.batch_size = 1 self.batch_output = [] - def predict(self, network, data): + def predict(self, data, seq_len_field_name=None): """Perform inference using the trained model. - :param network: a PyTorch model (cpu) :param data: a DataSet object. + :param str seq_len_field_name: field name indicating sequence lengths :return: list of batch outputs """ - # turn on the testing mode; clean up the history - self.mode(network, test=True) - batch_output = [] + if not isinstance(data, DataSet): + raise ValueError("Only Dataset class is allowed, not {}.".format(type(data))) + if seq_len_field_name is not None and seq_len_field_name not in data.field_arrays: + raise ValueError("Field name {} not found in DataSet {}.".format(seq_len_field_name, data)) - data_iterator = Batch(data, batch_size=self.batch_size, sampler=SequentialSampler(), as_numpy=False) + self.network.eval() + batch_output = defaultdict(list) + data_iterator = Batch(data, batch_size=self.batch_size, sampler=SequentialSampler(), as_numpy=False, + prefetch=False) - for batch_x, _ in data_iterator: - with torch.no_grad(): - prediction = self.data_forward(network, batch_x) - batch_output.append(prediction) + if hasattr(self.network, "predict"): + predict_func = self.network.predict + else: + predict_func = self.network.forward - return batch_output + with torch.no_grad(): + for batch_x, _ in data_iterator: + refined_batch_x = _build_args(predict_func, **batch_x) + prediction = predict_func(**refined_batch_x) - def mode(self, network, test=True): - if test: - network.eval() - else: - network.train() + if seq_len_field_name is not None: + seq_lens = batch_x[seq_len_field_name].tolist() + + for key, value in prediction.items(): + value = value.cpu().numpy() + if len(value.shape) == 1 or (len(value.shape) == 2 and value.shape[1] == 1): + batch_output[key].extend(value.tolist()) + else: + if seq_len_field_name is not None: + tmp_batch = [] + for idx, seq_len in enumerate(seq_lens): + tmp_batch.append(value[idx, :seq_len]) + batch_output[key].extend(tmp_batch) + else: + batch_output[key].append(value) - def data_forward(self, network, x): - """Forward through network.""" - y = network(**x) - return y + return batch_output diff --git a/fastNLP/core/sampler.py b/fastNLP/core/sampler.py index 67ec2a8d..c5784f59 100644 --- a/fastNLP/core/sampler.py +++ b/fastNLP/core/sampler.py @@ -1,89 +1,93 @@ +""" +sampler 子类实现了 fastNLP 所需的各种采样器。 +""" +__all__ = [ + "Sampler", + "BucketSampler", + "SequentialSampler", + "RandomSampler" +] + from itertools import chain import numpy as np -import torch - -def convert_to_torch_tensor(data_list, use_cuda): - """Convert lists into (cuda) Tensors. - :param data_list: 2-level lists - :param use_cuda: bool, whether to use GPU or not - :return data_list: PyTorch Tensor of shape [batch_size, max_seq_len] +class Sampler(object): """ - data_list = torch.Tensor(data_list).long() - if torch.cuda.is_available() and use_cuda: - data_list = data_list.cuda() - return data_list + 别名：:class:`fastNLP.Sampler` :class:`fastNLP.core.sampler.Sampler` + + `Sampler` 类的基类. 规定以何种顺序取出data中的元素 -class BaseSampler(object): - """The base class of all samplers. - - Sub-classes must implement the ``__call__`` method. - ``__call__`` takes a DataSet object and returns a list of int - the sampling indices. + 子类必须实现 ``__call__`` 方法. 输入 `DataSet` 对象, 返回其中元素的下标序列 """ - - def __call__(self, *args, **kwargs): + + def __call__(self, data_set): + """ + :param DataSet data_set: `DataSet` 对象, 需要Sample的数据 + :return result: list(int) 其中元素的下标序列, ``data_set`` 中元素会按 ``result`` 中顺序取出 + """ raise NotImplementedError -class SequentialSampler(BaseSampler): - """Sample data in the original order. +class SequentialSampler(Sampler): + """ + 别名：:class:`fastNLP.SequentialSampler` :class:`fastNLP.core.sampler.SequentialSampler` + + 顺序取出元素的 `Sampler` """ + def __call__(self, data_set): - """ - - :param DataSet data_set: - :return result: a list of integers. - """ return list(range(len(data_set))) -class RandomSampler(BaseSampler): - """Sample data in random permutation order. +class RandomSampler(Sampler): + """ + 别名：:class:`fastNLP.RandomSampler` :class:`fastNLP.core.sampler.RandomSampler` + + 随机化取元素的 `Sampler` """ + def __call__(self, data_set): - """ - - :param DataSet data_set: - :return result: a list of integers. - """ return list(np.random.permutation(len(data_set))) -class BucketSampler(BaseSampler): +class BucketSampler(Sampler): """ + 别名：:class:`fastNLP.BucketSampler` :class:`fastNLP.core.sampler.BucketSampler` - :param int num_buckets: the number of buckets to use. - :param int batch_size: batch size per epoch. - :param str seq_lens_field_name: the field name indicating the field about sequence length. + 带Bucket的 `Random Sampler`. 可以随机地取出长度相似的元素 + :param int num_buckets: bucket的数量 + :param int batch_size: batch的大小 + :param str seq_len_field_name: 对应序列长度的 `field` 的名字 """ - def __init__(self, num_buckets=10, batch_size=32, seq_lens_field_name='seq_lens'): + + def __init__(self, num_buckets=10, batch_size=32, seq_len_field_name='seq_len'): self.num_buckets = num_buckets self.batch_size = batch_size - self.seq_lens_field_name = seq_lens_field_name - + self.seq_len_field_name = seq_len_field_name + def __call__(self, data_set): - - seq_lens = data_set.get_all_fields()[self.seq_lens_field_name].content + seq_lens = data_set.get_all_fields()[self.seq_len_field_name].content total_sample_num = len(seq_lens) - + bucket_indexes = [] + assert total_sample_num >= self.num_buckets, "The number of samples is smaller than the number of buckets." num_sample_per_bucket = total_sample_num // self.num_buckets for i in range(self.num_buckets): bucket_indexes.append([num_sample_per_bucket * i, num_sample_per_bucket * (i + 1)]) bucket_indexes[-1][1] = total_sample_num - + sorted_seq_lens = list(sorted([(idx, seq_len) for idx, seq_len in zip(range(total_sample_num), seq_lens)], key=lambda x: x[1])) - + batchs = [] - + left_init_indexes = [] for b_idx in range(self.num_buckets): start_idx = bucket_indexes[b_idx][0] @@ -98,7 +102,7 @@ class BucketSampler(BaseSampler): if (left_init_indexes) != 0: batchs.append(left_init_indexes) np.random.shuffle(batchs) - + return list(chain(*batchs)) @@ -136,10 +140,10 @@ def k_means_1d(x, k, max_iter=100): if len(sorted_x) < k: raise ValueError("too few buckets") gap = len(sorted_x) / k - + centroids = np.array([sorted_x[int(x * gap)] for x in range(k)]) assign = None - + for i in range(max_iter): # Cluster Assignment step assign = np.array([np.argmin([np.absolute(x_i - x) for x in centroids]) for x_i in x]) @@ -171,7 +175,7 @@ def k_means_bucketing(lengths, buckets): bucket_data = [[] for _ in buckets] num_buckets = len(buckets) _, assignments = k_means_1d(lengths, num_buckets) - + for idx, bucket_id in enumerate(assignments): if buckets[bucket_id] is None or lengths[idx] <= buckets[bucket_id]: bucket_data[bucket_id].append(idx) diff --git a/fastNLP/core/tester.py b/fastNLP/core/tester.py index 48e1f090..883e0d01 100644 --- a/fastNLP/core/tester.py +++ b/fastNLP/core/tester.py @@ -1,50 +1,109 @@ +""" +tester模块实现了 fastNLP 所需的Tester类，能在提供数据、模型以及metric的情况下进行性能测试。 + +Example:: + + import numpy as np + import torch + from torch import nn + from fastNLP import Tester + from fastNLP import DataSet + from fastNLP import AccuracyMetric + + class Model(nn.Module): + def __init__(self): + super().__init__() + self.fc = nn.Linear(1, 1) + def forward(self, a): + return {'pred': self.fc(a.unsqueeze(1)).squeeze(1)} + + model = Model() + + dataset = DataSet({'a': np.arange(10, dtype=float), 'b':np.arange(10, dtype=float)*2}) + + dataset.set_input('a') + dataset.set_target('b') + + tester = Tester(dataset, model, metrics=AccuracyMetric()) + eval_results = tester.test() + +这里Metric的映射规律是和 :class:`fastNLP.Trainer` 中一致的，具体使用请参考 :doc:`trainer 模块` 的1.3部分。 +Tester在验证进行之前会调用model.eval()提示当前进入了evaluation阶段，即会关闭nn.Dropout()等，在验证结束之后会调用model.train()恢复到训练状态。 + + +""" +import warnings + import torch -from torch import nn +import torch.nn as nn + +from .batch import Batch +from .dataset import DataSet +from .metrics import _prepare_metrics +from .sampler import SequentialSampler +from .utils import _CheckError +from .utils import _build_args +from .utils import _check_loss_evaluate +from .utils import _move_dict_value_to_device +from .utils import _get_func_signature +from .utils import _get_model_device +from .utils import _move_model_to_device -from fastNLP.core.batch import Batch -from fastNLP.core.dataset import DataSet -from fastNLP.core.metrics import _prepare_metrics -from fastNLP.core.sampler import SequentialSampler -from fastNLP.core.utils import CheckError -from fastNLP.core.utils import _build_args -from fastNLP.core.utils import _check_loss_evaluate -from fastNLP.core.utils import _move_dict_value_to_device -from fastNLP.core.utils import get_func_signature +__all__ = [ + "Tester" +] class Tester(object): - """An collection of model inference and evaluation of performance, used over validation/dev set and test set. + """ + 别名：:class:`fastNLP.Tester` :class:`fastNLP.core.tester.Tester` - :param DataSet data: a validation/development set - :param torch.nn.modules.module model: a PyTorch model - :param MetricBase metrics: a metric object or a list of metrics (List[MetricBase]) - :param int batch_size: batch size for validation - :param bool use_cuda: whether to use CUDA in validation. - :param int verbose: the number of steps after which an information is printed. + Tester是在提供数据，模型以及metric的情况下进行性能测试的类。需要传入模型，数据以及metric进行验证。 - """ + :param data: 需要测试的数据集， :class:`~fastNLP.DataSet` 类型 + :param torch.nn.module model: 使用的模型 + :param metrics: :class:`~fastNLP.core.metrics.MetricBase` 或者一个列表的 :class:`~fastNLP.core.metrics.MetricBase` + :param int batch_size: evaluation时使用的batch_size有多大。 + :param str,int,torch.device,list(int) device: 将模型load到哪个设备。默认为None，即Trainer不对模型 + 的计算位置进行管理。支持以下的输入: - def __init__(self, data, model, metrics, batch_size=16, use_cuda=False, verbose=1): - super(Tester, self).__init__() + 1. str: ['cpu', 'cuda', 'cuda:0', 'cuda:1', ...] 依次为'cpu'中, 可见的第一个GPU中, 可见的第一个GPU中, + 可见的第二个GPU中; + + 2. torch.device：将模型装载到torch.device上。 + 3. int: 将使用device_id为该值的gpu进行训练 + + 4. list(int)：如果多于1个device，将使用torch.nn.DataParallel包裹model, 并使用传入的device。 + + 5. None. 为None则不对模型进行任何处理，如果传入的model为torch.nn.DataParallel该值必须为None。 + + 如果模型是通过predict()进行预测的话，那么将不能使用多卡(DataParallel)进行验证，只会使用第一张卡上的模型。 + :param int verbose: 如果为0不输出任何信息; 如果为1，打印出验证结果。 + """ + + def __init__(self, data, model, metrics, batch_size=16, device=None, verbose=1): + super(Tester, self).__init__() + if not isinstance(data, DataSet): raise TypeError(f"The type of data must be `fastNLP.DataSet`, got `{type(data)}`.") if not isinstance(model, nn.Module): raise TypeError(f"The type of model must be `torch.nn.Module`, got `{type(model)}`.") - + self.metrics = _prepare_metrics(metrics) - + self.data = data - self.use_cuda = use_cuda + self._model = _move_model_to_device(model, device=device) self.batch_size = batch_size self.verbose = verbose - - if torch.cuda.is_available() and self.use_cuda: - self._model = model.cuda() - else: - self._model = model - self._model_device = model.parameters().__next__().device - + + # 如果是DataParallel将没有办法使用predict方法 + if isinstance(self._model, nn.DataParallel): + if hasattr(self._model.module, 'predict') and not hasattr(self._model, 'predict'): + warnings.warn("Cannot use DataParallel to test your model, because your model offer predict() function," + " while DataParallel has no predict() function.") + self._model = self._model.module + # check predict if hasattr(self._model, 'predict'): self._predict_func = self._model.predict @@ -54,14 +113,15 @@ class Tester(object): f"for evaluation, not `{type(self._predict_func)}`.") else: self._predict_func = self._model.forward - + def test(self): - """Start test or validation. - - :return eval_results: a dictionary whose keys are the class name of metrics to use, values are the evaluation results of these metrics. + """开始进行验证，并返回验证结果。 + :return Dict[Dict] : dict的二层嵌套结构，dict的第一层是metric的名称; 第二层是这个metric的指标。 + 一个AccuracyMetric的例子为{'AccuracyMetric': {'acc': 1.0}}。 """ # turn on the testing mode; clean up the history + self._model_device = _get_model_device(self._model) network = self._model self._mode(network, is_test=True) data_iterator = Batch(self.data, self.batch_size, sampler=SequentialSampler(), as_numpy=False) @@ -72,28 +132,28 @@ class Tester(object): _move_dict_value_to_device(batch_x, batch_y, device=self._model_device) pred_dict = self._data_forward(self._predict_func, batch_x) if not isinstance(pred_dict, dict): - raise TypeError(f"The return value of {get_func_signature(self._predict_func)} " + raise TypeError(f"The return value of {_get_func_signature(self._predict_func)} " f"must be `dict`, got {type(pred_dict)}.") for metric in self.metrics: metric(pred_dict, batch_y) for metric in self.metrics: eval_result = metric.get_metric() if not isinstance(eval_result, dict): - raise TypeError(f"The return value of {get_func_signature(metric.get_metric)} must be " + raise TypeError(f"The return value of {_get_func_signature(metric.get_metric)} must be " f"`dict`, got {type(eval_result)}") metric_name = metric.__class__.__name__ eval_results[metric_name] = eval_result - except CheckError as e: - prev_func_signature = get_func_signature(self._predict_func) + except _CheckError as e: + prev_func_signature = _get_func_signature(self._predict_func) _check_loss_evaluate(prev_func_signature=prev_func_signature, func_signature=e.func_signature, check_res=e.check_res, pred_dict=pred_dict, target_dict=batch_y, dataset=self.data, check_level=0) - + if self.verbose >= 1: print("[tester] \n{}".format(self._format_eval_results(eval_results))) self._mode(network, is_test=False) return eval_results - + def _mode(self, model, is_test=False): """Train mode or Test mode. This is for PyTorch currently. @@ -105,13 +165,13 @@ class Tester(object): model.eval() else: model.train() - + def _data_forward(self, func, x): """A forward pass of the model. """ x = _build_args(func, **x) y = func(**x) return y - + def _format_eval_results(self, results): """Override this method to support more print formats. diff --git a/fastNLP/core/trainer.py b/fastNLP/core/trainer.py index 109315a3..2523a957 100644 --- a/fastNLP/core/trainer.py +++ b/fastNLP/core/trainer.py @@ -1,85 +1,428 @@ +r""" +Trainer在fastNLP中用于组织单任务的训练过程，可以避免用户在不同训练任务中重复撰以下步骤的代码 + + (1) epoch循环; + + (2) 将数据分成不同的Batch; + + (3) 对Batch进行pad; + + (4) 每个epoch结束或一定step后进行验证集验证; + + (5) 保存获得更好验证性能的模型。 + +1 Trainer的基本使用 + 下面的例子是使用神经网络来进行预测一个序列中是否有偶数个1。 + + Example:: + + import numpy as np + from torch import nn + import torch + import torch.nn.functional as F + from torch.optim import SGD + + from fastNLP import DataSet + from fastNLP import Trainer + from fastNLP import CrossEntropyLoss + from fastNLP import AccuracyMetric + from fastNLP.modules.decoder import MLP + + # 模型 + class Model(nn.Module): + def __init__(self, input_num): + super().__init__() + self.fcs = MLP([input_num, 40, 40, 2], 'relu') + + def forward(self, x): + x = self.fcs(x) + return {'pred': x} + model = Model(10) + + # 生成数据 + def generate_psedo_dataset(num_samples): + dataset = DataSet() + data = np.random.randint(2, size=(num_samples, 10)) + label = np.sum(data, axis=1)%2 + dataset = DataSet({'x':data.astype(float), 'label': label}) + dataset.set_input('x') + dataset.set_target('label') + return dataset + tr_dataset = generate_psedo_dataset(1000) + dev_data = generate_psedo_dataset(100) + + # 训练 + trainer = Trainer(tr_dataset, model, loss=CrossEntropyLoss(target='label'), + optimizer=SGD(model.parameters(), lr=0.1),n_epochs=1000, + dev_data = dev_data, metrics=AccuracyMetric(target='label')) + trainer.train() + + 由上面的例子可以看出通过使用Trainer，可以使得训练部分的代码大幅减少。 + 使用Trainer需要满足以下几个条件: + +1.1 模型 + 1 模型的forward()的参数名需要与DataSet中的名字对应。实际上fastNLP在将DataSet中的数据传递给模型forward()时，是 + 通过匹配名称实现的。所以上例中，如果Model的forward函数修改为forward(self, data), 则DataSet中的'x'这个field就应该 + 改名为'data'。 + + 2 传递给forward()的参数是DataSet中被设置为input的那些field。但如果forward()中没有对应的参数，则不会将数据传递 + 给forward()。例如，DataSet中'x1', 'x2'都是input，但是模型的函数为forward(self, x1), 那么'x2'不会传递给forward()。 + + 3 模型的forward()返回值需要为一个dict。 + +1.2 Loss + fastNLP中的为了不限制forward函数的返回内容数量(比如一些复杂任务需要返回多个内容，如Dependency Parsing， + :mod:`Loss` 与 :mod:`Metric` 都使用了通过名称来匹配相应内容的策略。如上面的例子中 + + Example:: + + trainer = Trainer(tr_dataset, model, loss=CrossEntropyLoss(target='label'), + optimizer=SGD(model.parameters(), lr=0.1),n_epochs=1000, + dev_data = dev_data, metrics=AccuracyMetric(target='label')) + + loss被设置为了 :class:`~fastNLP.CrossEntropyLoss` , 但在初始化的时候传入了target='label'这个参数， + :class:`~fastNLP.CrossEntropyLoss` 的初始化参数为(pred=None, target=None, padding_idx=-100)。 + + 这里的两个参数分别为计算CrossEntropy时需要使用到的模型的预测值与真实值。 + 其中 `pred` 一般来自于模型forward()的返回结果，`target` 一般是来自于DataSet中被设置为target的field。 + 由于每个人对真实值或者model的返回值取名并不一样，所以fastNLP的 :mod:`Loss` 提供一种类似于映射的机制来匹配对应的值， + 比如这里 :class:`~fastNLP.CrossEntropyLoss` 将尝试找到名为'label'的内容来作为真实值得到loss； + 而pred=None, 则 :class:`~fastNLP.CrossEntropyLoss` 使用'pred'作为名称匹配预测值， + 正好forward的返回值也叫pred，所以这里不需要申明pred。 + + 尽管fastNLP使用了映射机制来使得loss的计算变得比较灵活，但有些情况下loss必须在模型中进行计算，比如使用了CRF的模型。 + fastNLP中提供了 :class:`~fastNLP.LossInForward` 这个loss。 + 这个loss的原理是直接在forward()的返回结果中找到loss_key(默认寻找'loss')指定的那个tensor，并使用它作为loss。 + 如果Trainer初始化没有提供loss则默认使用 :class:`~fastNLP.LossInForward` 。 + + .. todo:: + 补充一个例子 详细例子可以参照 + +1.3 Metric + :mod:`Metric` 使用了与上述Loss一样的策略，即使用名称进行匹配。 + AccuracyMetric(target='label')的情况与CrossEntropyLoss 是同理的。 + + 在进行验证时，可能用到的计算与forward()中不太一致，没有办法直接从forward()的结果中得到预测值，这时模型可以提供一个predict()方法， + 如果提供的模型具有predict方法，则在模型验证时将调用predict()方法获取预测结果， + 传入到predict()的参数也是从DataSet中被设置为input的field中选择出来的; + 与forward()一样，返回值需要为一个dict。 + + .. todo:: + 补充一个例子 具体例子可以参考 + +2 Trainer的代码检查 + 由于在fastNLP中采取了映射的机制，所以难免可能存在对应出错的情况。Trainer提供一种映射检查机制，可以通过check_code_level来进行控制 + 比如下面的例子中，由于各种原因产生的报错 + +Example2.1 + :: + + import numpy as np + from torch import nn + import torch + from torch.optim import SGD + from fastNLP import Trainer + from fastNLP import DataSet + + class Model(nn.Module): + def __init__(self): + super().__init__() + self.fc = nn.Linear(1, 1) + def forward(self, x, b): + loss = torch.mean((self.fc(x)-b)**2) + return {'loss': loss} + model = Model() + + dataset = DataSet({'a': np.arange(10), 'b':np.arange(10)*2}) + dataset.set_input('a', 'b') + + trainer = Trainer(dataset, model, loss=None, optimizer=SGD(model.parameters(), lr=0.001)) + + trainer = Trainer(dataset, model, SGD(model.parameters())) + # 会报以下的错误 + # input fields after batch(if batch size is 2): + # a: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + # b: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) + # There is no target field. + # .... + # NameError: + # Problems occurred when calling Model.forward(self, x, b) + # missing param: ['x'] + # unused field: ['a'] + # Suggestion: You need to provide ['x'] in DataSet and set it as input. + + 这里就是由于在Trainer初始化的时候，fastNLP会尝试使用一个batch_size=2的batch去运行一遍forward()以及backward()。这里有两类 + 信息可以为你提供参考 + + 1 'input fields after batch...'这部分显示的是train dataset经过Batch操作后，每个field对应的类型以及进行shape。这里 + 因为train dataset没有target所以没有显示。根据这里可以看出是否正确将需要的内容设置为了input或target。 + + 2 NameError，NameError发生在映射出错的情况。这里报错的原因是由于尝试进行forward计算时(可以通过Model.forward(self, x, b)判断 + 出当前是在调取forward)，却没有获取到forward()函数中需要的'x'；在报错信息中同时指出了缺'x'，而'a'没有被使用，那么可能 + 就是由于field的名称不对。这里将dataset中'a'这个field的名称改为'x'，或者model的参数从'x'修改为'a'都可以解决问题。 + + 下面的例子是由于loss计算的时候找不到需要的值 + +Example2.2 + :: + + import numpy as np + from torch import nn + from torch.optim import SGD + from fastNLP import Trainer + from fastNLP import DataSet + from fastNLP import L1Loss + import torch + + class Model(nn.Module): + def __init__(self): + super().__init__() + self.fc = nn.Linear(1, 1) + def forward(self, a): + return {'pred_b': self.fc(a.unsqueeze(1)).squeeze(1), 'No use':1} + + model = Model() + + dataset = DataSet({'a': np.arange(10, dtype=float), 'b':np.arange(10, dtype=float)*2}) + + dataset.set_input('a') + dataset.set_target('b') + + trainer = Trainer(dataset, model, loss=L1Loss(target='label'), optimizer=SGD(model.parameters(), lr=0.001)) + # 报错信息如下 + # input fields after batch(if batch size is 2): + # a: (1)type:torch.Tensor (2)dtype:torch.float32, (3)shape:torch.Size([2]) + # target fields after batch(if batch size is 2): + # b: (1)type:torch.Tensor (2)dtype:torch.float32, (3)shape:torch.Size([2]) + # .... + # NameError: + # Problems occurred when calling L1Loss.get_loss(self, pred, target) + # missing param: ['pred(assign to `pred` in `L1Loss`)', 'label(assign to `target` in `L1Loss`)'] + # unused field: ['b'] + # unused param: ['pred_b', 'No use'] + # target field: ['b'] + # param from Model.forward(self, a): ['pred_b', 'No use'] + # Suggestion: (1). Check key assignment for `target` when initialize L1Loss. Or provide `label` in DataSet or output of Model.forward(self, a). + # (2). Check key assignment for `pred` when initialize L1Loss. Or provide `pred` in DataSet or output of Model.forward(self, a). + + 报错信息也包含两部分: + + 1 第一部分与上面是一样的 + + 2 这里报错的原因是由于计算loss的时候找不到相应的值(通过L1Loss.get_loss(self, pred, target)判断出来的)； + 报错的原因是因为 `pred` 和 `label` (我们在初始化L1Loss时将target指定为了label)都没有找到。 + 这里'unused field'是DataSet中出现了，但却没有被设置为input或者target的field； + 'unused param'是forward()中返回且没有被使用到的内容；'target field'是被设置为了target的field; + 'param from Model.forward(self, a)'是forward()返回的所有key。"Suggestion"是关于当前错误处理的建议。 + + 但是在一些情况下，比如forward()返回值只有一个，target也只有一个，fastNLP不会进行匹配，而直接将forward()的结果作为pred, + 将DataSet中的target设置为target。上面的例子在返回值中加入了一个'No use'则只是为了使得Loss去匹配结果。 + + + 下面是带有dev dataset时如果出现错误会发生的报错， + +Example2.3 + :: + + import numpy as np + from torch import nn + from torch.optim import SGD + from fastNLP import Trainer + from fastNLP import DataSet + from fastNLP import AccuracyMetric + import torch + + class Model(nn.Module): + def __init__(self): + super().__init__() + self.fc = nn.Linear(1, 1) + def forward(self, a, b): + loss = torch.mean((self.fc(a.float().unsqueeze(1))-b.float())**2) + return {'loss': loss} + def predict(self, a): # 使用predict()进行验证 + return {'output':self.fc(a.float().unsqueeze(1))} #这里return的值不包含'pred'这个key + model = Model() + + dataset = DataSet({'a': np.arange(10), 'b':np.arange(10)*2}) + dev_data = DataSet({'a': np.arange(10, 20), 'b':np.arange(10, 20)*2}) + + dataset.set_input('a', 'b') + dev_data.set_input('a') # 这里没有设置target + + trainer = Trainer(dataset, model, loss=None, optimizer=SGD(model.parameters(), lr=0.001), + dev_data=dev_data, metrics=AccuracyMetric()) + + # 报错信息 + # ... + # NameError: + # Problems occurred when calling AccuracyMetric.evaluate(self, pred, target, seq_len=None) + # missing param: ['pred(assign to `pred` in `AccuracyMetric`)', 'target(assign to `target` in `AccuracyMetric`)'] + # unused param: ['output'] + # target field: [] + # param from Model.predict(self, a): ['output'] + # Suggestion: (1). Check key assignment for `pred` when initialize AccuracyMetric. Or provide `pred` in DataSet or output of Model.predict(self, a). + # (2). Check key assignment for `target` when initialize AccuracyMetric. Or provide `target` in DataSet or output of Model.predict(self, a). + + 报错信息和前面都是类似的，但是可以通过'AccuracyMetric.evaluate(self, pred, target, seq_len=None)'看出这里是evaluation + 的时候发生了错误。这样避免了需要在完成一整个epoch的训练才能发现evaluation弄错的情况。这里的修改是通过在初始化metric的时候 + 指明通过'output'获取`pred`, 即AccuracyMetric(pred='output')。 + + 可以通过check_code_level调节检查的强度。默认为0，即进行检查。 + +3 Trainer与callback + 虽然Trainer本身已经集成了一些功能，但仍然不足以囊括训练过程中可能需要到的功能，比如负采样，learning rate decay, Early Stop等。 + 为了解决这个问题fastNLP引入了callback的机制，:class:`~fastNLP.Callback` 是一种在Trainer训练过程中特定阶段会运行的函数集合， + 所有的 :class:`~fastNLP.Callback` 都具有on_*(比如on_train_start, on_backward_begin)等函数。 + 如果 Callback 实现了该函数，则Trainer运行至对应阶段，会进行调用，例如:: + + from fastNLP import Callback, EarlyStopCallback, Trainer, CrossEntropyLoss, AccuracyMetric + from fastNLP.models import CNNText + + start_time = time.time() + + class MyCallback(Callback): + def on_epoch_end(self): + print('{:d}ms\n\n'.format(round((time.time()-start_time)*1000))) + + model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1) + trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, loss=CrossEntropyLoss(), + metrics=AccuracyMetric(), callbacks=[MyCallback(),EarlyStopCallback(10)]) + trainer.train() + + 这里，我们通过继承 :class:`~fastNLP.Callback` 类定义了自己的 callback 的，并和内置的 :class:`~fastNLP.EarlyStopCallback` + 一起传给了 :class:`~fastNLP.Trainer` ，增强了 :class:`~fastNLP.Trainer` 的功能 + + fastNLP已经自带了很多callback函数供使用，可以参考 :doc:`fastNLP.core.callback` 。 + +""" +__all__ = [ + "Trainer" +] + import os import time -from datetime import datetime -from datetime import timedelta +from datetime import datetime, timedelta import numpy as np import torch -from tensorboardX import SummaryWriter -from torch import nn +import torch.nn as nn try: - from tqdm.autonotebook import tqdm + from tqdm.auto import tqdm except: - from fastNLP.core.utils import pseudo_tqdm as tqdm - -from fastNLP.core.batch import Batch -from fastNLP.core.callback import CallbackManager -from fastNLP.core.dataset import DataSet -from fastNLP.core.losses import _prepare_losser -from fastNLP.core.metrics import _prepare_metrics -from fastNLP.core.optimizer import Adam -from fastNLP.core.sampler import BaseSampler -from fastNLP.core.sampler import RandomSampler -from fastNLP.core.sampler import SequentialSampler -from fastNLP.core.tester import Tester -from fastNLP.core.utils import CheckError -from fastNLP.core.utils import _build_args -from fastNLP.core.utils import _check_forward_error -from fastNLP.core.utils import _check_loss_evaluate -from fastNLP.core.utils import _move_dict_value_to_device -from fastNLP.core.utils import get_func_signature + from .utils import _pseudo_tqdm as tqdm + +from .batch import Batch +from .callback import CallbackManager, CallbackException +from .dataset import DataSet +from .losses import _prepare_losser +from .metrics import _prepare_metrics +from .optimizer import Optimizer +from .sampler import Sampler +from .sampler import RandomSampler +from .sampler import SequentialSampler +from .tester import Tester +from .utils import _CheckError +from .utils import _build_args +from .utils import _check_forward_error +from .utils import _check_loss_evaluate +from .utils import _move_dict_value_to_device +from .utils import _get_func_signature +from .utils import _get_model_device +from .utils import _move_model_to_device class Trainer(object): - def __init__(self, train_data, model, loss=None, metrics=None, n_epochs=3, batch_size=32, print_every=50, - validate_every=-1, dev_data=None, save_path=None, optimizer=Adam(lr=0.01, weight_decay=0), - check_code_level=0, metric_key=None, sampler=RandomSampler(), use_tqdm=True, use_cuda=False, - callbacks=None): - """ - :param DataSet train_data: the training data - :param torch.nn.modules.module model: a PyTorch model - :param LossBase loss: a loss object - :param MetricBase metrics: a metric object or a list of metrics (List[MetricBase]) - :param int n_epochs: the number of training epochs - :param int batch_size: batch size for training and validation - :param int print_every: step interval to print next training information. Default: -1(no print). - :param int validate_every: step interval to do next validation. Default: -1(validate every epoch). - :param DataSet dev_data: the validation data - :param bool use_cuda: whether to use CUDA in training. - :param str save_path: file path to save models - :param Optimizer optimizer: an optimizer object - :param int check_code_level: level of FastNLP code checker. -1: don't check, 0: ignore. 1: warning. 2: strict.\\ - `ignore` will not check unused field; `warning` when warn if some field are not used; `strict` means - it will raise error if some field are not used. - :param str metric_key: a single indicator used to decide the best model based on metric results. It must be one - of the keys returned by the FIRST metric in `metrics`. If the overall result gets better if the indicator gets - smaller, add "-" in front of the string. For example:: - - metric_key="-PPL" # language model gets better as perplexity gets smaller - :param BaseSampler sampler: method used to generate batch data. - :param bool use_tqdm: whether to use tqdm to show train progress. - - """ + """ + 别名：:class:`fastNLP.Trainer` :class:`fastNLP.core.trainer.Trainer` + + Trainer在fastNLP中用于组织单任务的训练过程，可以避免用户在不同训练任务中重复撰写 + (1) epoch循环; + (2) 将数据分成不同的Batch; + (3) 对Batch进行pad; + (4) 每个epoch结束或一定step后进行验证集验证; + (5) 保存获得更好验证性能的模型等。 + + 详细的介绍参见 :doc:`fastNLP.core.trainer` + + :param train_data: 训练集， :class:`~fastNLP.DataSet` 类型。 + :param nn.modules model: 待训练的模型 + :param optimizer: `torch.optim.Optimizer` 优化器。如果为None，则Trainer使用默认的Adam(model.parameters(), lr=4e-3)这个优化器 + :param int batch_size: 训练和验证的时候的batch大小。 + :param loss: 使用的 :class:`~fastNLP.core.losses.LossBase` 对象。当为None时，默认使用 :class:`~fastNLP.LossInForward` + :param sampler: Batch数据生成的顺序， :class:`~fastNLP.Sampler` 类型。如果为None，默认使用 :class:`~fastNLP.RandomSampler` + :param update_every: int, 多少步更新一次梯度。用于希望累计梯度的场景，比如需要128的batch_size, 但是直接设为128 + 会导致内存不足，通过设置batch_size=32, update_every=4达到目的。当optimizer为None时，该参数无效。 + :param int n_epochs: 需要优化迭代多少次。 + :param int print_every: 多少次反向传播更新tqdm显示的loss; 如果use_tqdm=False, 则多少次反向传播打印loss。 + :param dev_data: 用于做验证的DataSet， :class:`~fastNLP.DataSet` 类型。 + :param metrics: 验证的评估函数。可以只使用一个 :class:`Metric` ， + 也可以使用多个 :class:`Metric` ，通过列表传入。 + 如验证时取得了更好的验证结果(如果有多个Metric，以列表中第一个Metric为准)，且save_path不为None， + 则保存当前模型。Metric种类详见 :doc:`metrics模块 ` 。仅在传入dev_data时有效。 + :param str,None metric_key: :class:`Metric` 有时会有多个指标， + 比如 :class:`~fastNLP.core.metrics.SpanFPreRecMetric` 中包含了'f', 'pre', 'rec'。此时需 + 要指定以哪个指标为准。另外有些指标是越小效果越好，比如语言模型的困惑度，这种情况下，在key前面增加一个'-'来表 + 明验证时，值越小越好(比如: "-ppl")。仅在传入dev_data时有效。 + :param int validate_every: 多少个step在验证集上验证一次; 如果为-1，则每个epoch结束验证一次。仅在传入dev_data时有效。 + :param str,None save_path: 将模型保存路径。如果为None，则不保存模型。如果dev_data为None，则保存最后一次迭代的模型。 + 保存的时候不仅保存了参数，还保存了模型结构。即便使用DataParallel，这里也只保存模型。 + :param prefetch: bool, 是否使用额外的进程对产生batch数据。理论上会使得Batch迭代更快。 + :param bool use_tqdm: 是否使用tqdm来显示训练进度; 如果为False，则将loss打印在终端中。 + :param str,int,torch.device,list(int) device: 将模型load到哪个设备。默认为None，即Trainer不对模型 + 的计算位置进行管理。支持以下的输入: + + 1. str: ['cpu', 'cuda', 'cuda:0', 'cuda:1', ...] 依次为'cpu'中, 可见的第一个GPU中, 可见的第一个GPU中, + 可见的第二个GPU中; + + 2. torch.device：将模型装载到torch.device上。 + + 3. int: 将使用device_id为该值的gpu进行训练 + + 4. list(int)：如果多于1个device，将使用torch.nn.DataParallel包裹model, 并使用传入的device。 + + 5. None. 为None则不对模型进行任何处理，如果传入的model为torch.nn.DataParallel该值必须为None。 + + 已知可能会出现的问题：Adagrad优化器可能无法正常使用这个参数，请手动管理模型位置。 + + :param list(callbacks) callbacks: 用于在train过程中起调节作用的回调函数。比如early stop，negative sampling等可以 + 通过callback机制实现。 可使用的callback参见 :doc:`callback模块 ` + :param int check_code_level: 模型检查等级. -1: 不进行检查; 0: 仅出现错误时停止; 1: 如果有field没有被使用， + 报告警告信息; 2: 有任何field没有被使用都报错. 检查的原理是通过使用很小的batch(默认2个sample)来运行代码，但是 + 这个过程理论上不会修改任何参数，只是会检查能否运行。但如果(1)模型中存在将batch_size写为某个固定值的情况； + (2)模型中存在累加前向计算次数的，可能会多计算1次。以上情况建议将check_code_level设置为-1。 + """ + + def __init__(self, train_data, model, optimizer=None, loss=None, + batch_size=32, sampler=None, update_every=1, + n_epochs=10, print_every=5, + dev_data=None, metrics=None, metric_key=None, + validate_every=-1, save_path=None, + prefetch=False, use_tqdm=True, device=None, + callbacks=None, + check_code_level=0): super(Trainer, self).__init__() - if not isinstance(train_data, DataSet): raise TypeError(f"The type of train_data must be fastNLP.DataSet, got {type(train_data)}.") if not isinstance(model, nn.Module): raise TypeError(f"The type of model must be torch.nn.Module, got {type(model)}.") - + # check metrics and dev_data if (not metrics) and dev_data is not None: raise ValueError("No metric for dev_data evaluation.") if metrics and (dev_data is None): raise ValueError("No dev_data for evaluations, pass dev_data or set metrics to None. ") - + + # check update every + assert update_every >= 1, "update_every must be no less than 1." + self.update_every = int(update_every) + # check save_path if not (save_path is None or isinstance(save_path, str)): raise ValueError("save_path can only be None or `str`.") # prepare evaluate metrics = _prepare_metrics(metrics) - + # parse metric_key # increase_better is True. It means the exp result gets better if the indicator increases. # It is true by default. @@ -89,19 +432,20 @@ class Trainer(object): self.metric_key = metric_key[1:] if metric_key[0] == "+" or metric_key[0] == "-" else metric_key elif len(metrics) > 0: self.metric_key = metrics[0].__class__.__name__.lower().strip('metric') - + # prepare loss losser = _prepare_losser(loss) - + # sampler check - if not isinstance(sampler, BaseSampler): + if sampler is not None and not isinstance(sampler, Sampler): raise ValueError("The type of sampler should be fastNLP.BaseSampler, got {}.".format(type(sampler))) - + if check_code_level > -1: _check_code(dataset=train_data, model=model, losser=losser, metrics=metrics, dev_data=dev_data, metric_key=metric_key, check_level=check_code_level, batch_size=min(batch_size, DEFAULT_CHECK_BATCH_SIZE)) - + # _check_code 是 fastNLP 帮助你检查代码是否正确的方法 。如果你在错误栈中看到这行注释，请认真检查你的代码 + self.train_data = train_data self.dev_data = dev_data # If None, No validation. self.model = model @@ -109,102 +453,88 @@ class Trainer(object): self.metrics = metrics self.n_epochs = int(n_epochs) self.batch_size = int(batch_size) - self.use_cuda = bool(use_cuda) self.save_path = save_path self.print_every = int(print_every) - self.validate_every = int(validate_every) if validate_every!=0 else -1 + self.validate_every = int(validate_every) if validate_every != 0 else -1 self.best_metric_indicator = None - self.sampler = sampler - self.callback_manager = CallbackManager(env={"trainer": self}, callbacks=callbacks) - + self.best_dev_epoch = None + self.best_dev_step = None + self.best_dev_perf = None + self.sampler = sampler if sampler is not None else RandomSampler() + self.prefetch = prefetch + self.n_steps = (len(self.train_data) // self.batch_size + int( + len(self.train_data) % self.batch_size != 0)) * self.n_epochs + + self.model = _move_model_to_device(self.model, device=device) + if isinstance(optimizer, torch.optim.Optimizer): self.optimizer = optimizer + elif isinstance(optimizer, Optimizer): + self.optimizer = optimizer.construct_from_pytorch(model.parameters()) + elif optimizer is None: + self.optimizer = torch.optim.Adam(model.parameters(), lr=4e-3) else: - self.optimizer = optimizer.construct_from_pytorch(self.model.parameters()) - + raise TypeError("optimizer can only be torch.optim.Optimizer type, not {}.".format(type(optimizer))) + self.use_tqdm = use_tqdm + self.pbar = None self.print_every = abs(self.print_every) - + if self.dev_data is not None: self.tester = Tester(model=self.model, data=self.dev_data, metrics=self.metrics, batch_size=self.batch_size, - use_cuda=self.use_cuda, + device=None, # 由上面的部分处理device verbose=0) - + self.step = 0 self.start_time = None # start timestamp - + + self.callback_manager = CallbackManager(env={"trainer": self}, + callbacks=callbacks) + def train(self, load_best_model=True): """ + 使用该函数使Trainer开始训练。 + + :param bool load_best_model: 该参数只有在初始化提供了dev_data的情况下有效， + 如果True, trainer将在返回之前重新加载dev表现最好的模型参数。 + :return dict: 返回一个字典类型的数据, + 内含以下内容:: - 开始训练过程。主要有以下几个步骤:: - - for epoch in range(num_epochs): - # 使用Batch从DataSet中按批取出数据，并自动对DataSet中dtype为(float, int)的fields进行padding。并转换为Tensor。 - 非float，int类型的参数将不会被转换为Tensor，且不进行padding。 - for batch_x, batch_y in Batch(DataSet) - # batch_x是一个dict, 被设为input的field会出现在这个dict中， - key为DataSet中的field_name, value为该field的value - # batch_y也是一个dict，被设为target的field会出现在这个dict中， - key为DataSet中的field_name, value为该field的value - 2. 将batch_x的数据送入到model.forward函数中，并获取结果。这里我们就是通过匹配batch_x中的key与forward函数的形 - 参完成参数传递。例如， - forward(self, x, seq_lens) # fastNLP会在batch_x中找到key为"x"的value传递给x，key为"seq_lens"的 - value传递给seq_lens。若在batch_x中没有找到所有必须要传递的参数，就会报错。如果forward存在默认参数 - 而且默认参数这个key没有在batch_x中，则使用默认参数。 - 3. 将batch_y与model.forward的结果一并送入loss中计算loss。loss计算时一般都涉及到pred与target。但是在不同情况 - 中，可能pred称为output或prediction, target称为y或label。fastNLP通过初始化loss时传入的映射找到pred或 - target。比如在初始化Trainer时初始化loss为CrossEntropyLoss(pred='output', target='y'), 那么fastNLP计 - 算loss时，就会使用"output"在batch_y与forward的结果中找到pred;使用"y"在batch_y与forward的结果中找target - , 并完成loss的计算。 - 4. 获取到loss之后，进行反向求导并更新梯度 - 根据需要适时进行验证机测试 - 根据metrics进行evaluation，并根据是否提供了save_path判断是否存储模型 - - :param bool load_best_model: 该参数只有在初始化提供了dev_data的情况下有效，如果True, trainer将在返回之前重新加载dev表现 - 最好的模型参数。 - :return results: 返回一个字典类型的数据, 内含以下内容:: - - seconds: float, 表示训练时长 - 以下三个内容只有在提供了dev_data的情况下会有。 - best_eval: Dict of Dict, 表示evaluation的结果 - best_epoch: int，在第几个epoch取得的最佳值 - best_step: int, 在第几个step(batch)更新取得的最佳值 + seconds: float, 表示训练时长 + 以下三个内容只有在提供了dev_data的情况下会有。 + best_eval: Dict of Dict, 表示evaluation的结果。第一层的key为Metric的名称， + 第二层的key为具体的Metric + best_epoch: int，在第几个epoch取得的最佳值 + best_step: int, 在第几个step(batch)更新取得的最佳值 """ results = {} + if self.n_epochs <= 0: + print(f"training epoch is {self.n_epochs}, nothing was done.") + results['seconds'] = 0. + return results try: - if torch.cuda.is_available() and self.use_cuda: - self.model = self.model.cuda() - self._model_device = self.model.parameters().__next__().device - + self._model_device = _get_model_device(self.model) self._mode(self.model, is_test=False) - - self.start_time = str(datetime.now().strftime('%Y-%m-%d %H-%M-%S')) + self._load_best_model = load_best_model + self.start_time = str(datetime.now().strftime('%Y-%m-%d-%H-%M-%S')) start_time = time.time() print("training epochs started " + self.start_time, flush=True) - if self.save_path is None: - class psudoSW: - def __getattr__(self, item): - def pass_func(*args, **kwargs): - pass - - return pass_func - - self._summary_writer = psudoSW() - else: - path = os.path.join(self.save_path, 'tensorboard_logs_{}'.format(self.start_time)) - self._summary_writer = SummaryWriter(path) - - self.callback_manager.before_train() - self._train() - self.callback_manager.after_train(self.model) - - if self.dev_data is not None: - print("\nIn Epoch:{}/Step:{}, got best dev performance:".format(self.best_dev_epoch, self.best_dev_step) + - self.tester._format_eval_results(self.best_dev_perf),) + + try: + self.callback_manager.on_train_begin() + self._train() + self.callback_manager.on_train_end() + except (CallbackException, KeyboardInterrupt) as e: + self.callback_manager.on_exception(e) + + if self.dev_data is not None and hasattr(self, 'best_dev_perf'): + print( + "\nIn Epoch:{}/Step:{}, got best dev performance:".format(self.best_dev_epoch, self.best_dev_step) + + self.tester._format_eval_results(self.best_dev_perf), ) results['best_eval'] = self.best_dev_perf results['best_epoch'] = self.best_dev_epoch results['best_step'] = self.best_dev_step @@ -216,58 +546,57 @@ class Trainer(object): else: print("Fail to reload best model.") finally: - self._summary_writer.close() - del self._summary_writer + pass results['seconds'] = round(time.time() - start_time, 2) - + return results - + def _train(self): if not self.use_tqdm: - from fastNLP.core.utils import pseudo_tqdm as inner_tqdm + from fastNLP.core.utils import _pseudo_tqdm as inner_tqdm else: inner_tqdm = tqdm self.step = 0 + self.epoch = 0 start = time.time() - data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False) - total_steps = data_iterator.num_batches * self.n_epochs - with inner_tqdm(total=total_steps, postfix='loss:{0:<6.5f}', leave=False, dynamic_ncols=True) as pbar: + + with inner_tqdm(total=self.n_steps, postfix='loss:{0:<6.5f}', leave=False, dynamic_ncols=True) as pbar: + self.pbar = pbar avg_loss = 0 - for epoch in range(1, self.n_epochs+1): + data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + self.batch_per_epoch = data_iterator.num_batches + for epoch in range(1, self.n_epochs + 1): + self.epoch = epoch pbar.set_description_str(desc="Epoch {}/{}".format(epoch, self.n_epochs)) # early stopping - self.callback_manager.before_epoch(epoch, self.n_epochs) + self.callback_manager.on_epoch_begin() for batch_x, batch_y in data_iterator: + self.step += 1 + _move_dict_value_to_device(batch_x, batch_y, device=self._model_device) indices = data_iterator.get_batch_indices() # negative sampling; replace unknown; re-weight batch_y - self.callback_manager.before_batch(batch_x, batch_y, indices) - _move_dict_value_to_device(batch_x, batch_y, device=self._model_device) + self.callback_manager.on_batch_begin(batch_x, batch_y, indices) prediction = self._data_forward(self.model, batch_x) - + # edit prediction - self.callback_manager.before_loss(batch_y, prediction) - loss = self._compute_loss(prediction, batch_y) + self.callback_manager.on_loss_begin(batch_y, prediction) + loss = self._compute_loss(prediction, batch_y).mean() avg_loss += loss.item() - + loss = loss / self.update_every + # Is loss NaN or inf? requires_grad = False - self.callback_manager.before_backward(loss, self.model) + self.callback_manager.on_backward_begin(loss) self._grad_backward(loss) - # gradient clipping - self.callback_manager.after_backward(self.model) - + self.callback_manager.on_backward_end() + self._update() - # lr scheduler; lr_finder; one_cycle - self.callback_manager.after_step(self.optimizer) - - self._summary_writer.add_scalar("loss", loss.item(), global_step=self.step) - for name, param in self.model.named_parameters(): - if param.requires_grad: - self._summary_writer.add_scalar(name + "_mean", param.mean(), global_step=self.step) - # self._summary_writer.add_scalar(name + "_std", param.std(), global_step=self.step) - # self._summary_writer.add_scalar(name + "_grad_sum", param.sum(), global_step=self.step) - if (self.step+1) % self.print_every == 0: + self.callback_manager.on_step_end() + + if self.step % self.print_every == 0: + avg_loss = float(avg_loss) / self.print_every if self.use_tqdm: - print_output = "loss:{0:<6.5f}".format(avg_loss / self.print_every) + print_output = "loss:{0:<6.5f}".format(avg_loss) pbar.update(self.print_every) else: end = time.time() @@ -276,50 +605,45 @@ class Trainer(object): epoch, self.step, avg_loss, diff) pbar.set_postfix_str(print_output) avg_loss = 0 - self.step += 1 - # do nothing - self.callback_manager.after_batch() - + self.callback_manager.on_batch_end() + if ((self.validate_every > 0 and self.step % self.validate_every == 0) or - (self.validate_every < 0 and self.step % len(data_iterator)) == 0) \ + (self.validate_every < 0 and self.step % len(data_iterator) == 0)) \ and self.dev_data is not None: eval_res = self._do_validation(epoch=epoch, step=self.step) eval_str = "Evaluation at Epoch {}/{}. Step:{}/{}. ".format(epoch, self.n_epochs, self.step, - total_steps) + \ + self.n_steps) + \ self.tester._format_eval_results(eval_res) - pbar.write(eval_str) - - # if self.validate_every < 0 and self.dev_data: - # eval_res = self._do_validation(epoch=epoch, step=self.step) - # eval_str = "Epoch {}/{}. Step:{}/{}. ".format(epoch, self.n_epochs, self.step, total_steps) + \ - # self.tester._format_eval_results(eval_res) - # pbar.write(eval_str) - if epoch != self.n_epochs: - data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, - as_numpy=False) + pbar.write(eval_str + '\n') + + # ================= mini-batch end ==================== # + # lr decay; early stopping - self.callback_manager.after_epoch(epoch, self.n_epochs, self.optimizer) + self.callback_manager.on_epoch_end() + # =============== epochs end =================== # pbar.close() - + self.pbar = None + # ============ tqdm end ============== # + def _do_validation(self, epoch, step): + self.callback_manager.on_valid_begin() res = self.tester.test() - for name, metric in res.items(): - for metric_key, metric_val in metric.items(): - self._summary_writer.add_scalar("valid_{}_{}".format(name, metric_key), metric_val, - global_step=self.step) + + is_better_eval = False if self._better_eval_result(res): if self.save_path is not None: self._save_model(self.model, - "best_" + "_".join([self.model.__class__.__name__, self.metric_key, self.start_time])) - else: - self._best_model_states = {name:param.cpu().clone() for name, param in self.model.named_parameters()} + "best_" + "_".join([self.model.__class__.__name__, self.metric_key, self.start_time])) + elif self._load_best_model: + self._best_model_states = {name: param.cpu().clone() for name, param in self.model.named_parameters()} self.best_dev_perf = res self.best_dev_epoch = epoch self.best_dev_step = step + is_better_eval = True # get validation results; adjust optimizer - self.callback_manager.after_valid(res, self.metric_key, self.optimizer) + self.callback_manager.on_valid_end(res, self.metric_key, self.optimizer, is_better_eval) return res - + def _mode(self, model, is_test=False): """Train mode or Test mode. This is for PyTorch currently. @@ -331,20 +655,22 @@ class Trainer(object): model.eval() else: model.train() - + def _update(self): """Perform weight update on a model. """ - self.optimizer.step() - + if self.optimizer is not None and (self.step + 1) % self.update_every == 0: + self.optimizer.step() + def _data_forward(self, network, x): x = _build_args(network.forward, **x) y = network(**x) if not isinstance(y, dict): - raise TypeError(f"The return value of {get_func_signature(network.forward)} should be dict, got {type(y)}.") + raise TypeError( + f"The return value of {_get_func_signature(network.forward)} should be dict, got {type(y)}.") return y - + def _grad_backward(self, loss): """Compute gradient with link rules. @@ -352,9 +678,10 @@ class Trainer(object): For PyTorch, just do "loss.backward()" """ - self.model.zero_grad() + if self.step % self.update_every == 0: + self.model.zero_grad() loss.backward() - + def _compute_loss(self, predict, truth): """Compute loss given prediction and ground truth. @@ -363,15 +690,30 @@ class Trainer(object): :return: a scalar """ return self.losser(predict, truth) - + def _save_model(self, model, model_name, only_param=False): + """ 存储不含有显卡信息的state_dict或model + :param model: + :param model_name: + :param only_param: + :return: + """ if self.save_path is not None: - model_name = os.path.join(self.save_path, model_name) + model_path = os.path.join(self.save_path, model_name) + if not os.path.exists(self.save_path): + os.makedirs(self.save_path, exist_ok=True) + if isinstance(model, nn.DataParallel): + model = model.module if only_param: - torch.save(model.state_dict(), model_name) + state_dict = model.state_dict() + for key in state_dict: + state_dict[key] = state_dict[key].cpu() + torch.save(state_dict, model_path) else: - torch.save(model, model_name) - + model.cpu() + torch.save(model, model_path) + model.to(self._model_device) + def _load_model(self, model, model_name, only_param=False): # 返回bool值指示是否成功reload模型 if self.save_path is not None: @@ -380,13 +722,16 @@ class Trainer(object): states = torch.load(model_path) else: states = torch.load(model_path).state_dict() - model.load_state_dict(states) + if isinstance(model, nn.DataParallel): + model.module.load_state_dict(states) + else: + model.load_state_dict(states) elif hasattr(self, "_best_model_states"): model.load_state_dict(self._best_model_states) else: return False return True - + def _better_eval_result(self, metrics): """Check if the current epoch yields better validation results. @@ -414,6 +759,7 @@ class Trainer(object): DEFAULT_CHECK_BATCH_SIZE = 2 DEFAULT_CHECK_NUM_BATCH = 2 + def _get_value_info(_dict): # given a dict value, return information about this dict's value. Return list of str strs = [] @@ -430,27 +776,28 @@ def _get_value_info(_dict): strs.append(_str) return strs + def _check_code(dataset, model, losser, metrics, batch_size=DEFAULT_CHECK_BATCH_SIZE, dev_data=None, metric_key=None, check_level=0): # check get_loss 方法 model_devcie = model.parameters().__next__().device - + batch = Batch(dataset=dataset, batch_size=batch_size, sampler=SequentialSampler()) for batch_count, (batch_x, batch_y) in enumerate(batch): _move_dict_value_to_device(batch_x, batch_y, device=model_devcie) # forward check - if batch_count==0: + if batch_count == 0: info_str = "" input_fields = _get_value_info(batch_x) target_fields = _get_value_info(batch_y) - if len(input_fields)>0: + if len(input_fields) > 0: info_str += "input fields after batch(if batch size is {}):\n".format(batch_size) info_str += "\n".join(input_fields) info_str += '\n' else: raise RuntimeError("There is no input field.") - if len(target_fields)>0: + if len(target_fields) > 0: info_str += "target fields after batch(if batch size is {}):\n".format(batch_size) info_str += "\n".join(target_fields) info_str += '\n' @@ -458,14 +805,14 @@ def _check_code(dataset, model, losser, metrics, batch_size=DEFAULT_CHECK_BATCH_ info_str += 'There is no target field.' print(info_str) _check_forward_error(forward_func=model.forward, dataset=dataset, - batch_x=batch_x, check_level=check_level) - + batch_x=batch_x, check_level=check_level) + refined_batch_x = _build_args(model.forward, **batch_x) pred_dict = model(**refined_batch_x) - func_signature = get_func_signature(model.forward) + func_signature = _get_func_signature(model.forward) if not isinstance(pred_dict, dict): raise TypeError(f"The return value of {func_signature} should be `dict`, not `{type(pred_dict)}`.") - + # loss check try: loss = losser(pred_dict, batch_y) @@ -473,23 +820,23 @@ def _check_code(dataset, model, losser, metrics, batch_size=DEFAULT_CHECK_BATCH_ if batch_count == 0: if not isinstance(loss, torch.Tensor): raise TypeError( - f"The return value of {get_func_signature(losser.get_loss)} should be `torch.Tensor`, " + f"The return value of {_get_func_signature(losser.get_loss)} should be `torch.Tensor`, " f"but got `{type(loss)}`.") if len(loss.size()) != 0: raise ValueError( - f"The size of return value of {get_func_signature(losser.get_loss)} is {loss.size()}, " + f"The size of return value of {_get_func_signature(losser.get_loss)} is {loss.size()}, " f"should be torch.size([])") loss.backward() - except CheckError as e: - # TODO: another error raised if CheckError caught - pre_func_signature = get_func_signature(model.forward) + except _CheckError as e: + # TODO: another error raised if _CheckError caught + pre_func_signature = _get_func_signature(model.forward) _check_loss_evaluate(prev_func_signature=pre_func_signature, func_signature=e.func_signature, check_res=e.check_res, pred_dict=pred_dict, target_dict=batch_y, dataset=dataset, check_level=check_level) model.zero_grad() if batch_count + 1 >= DEFAULT_CHECK_NUM_BATCH: break - + if dev_data is not None: tester = Tester(data=dev_data[:batch_size * DEFAULT_CHECK_NUM_BATCH], model=model, metrics=metrics, batch_size=batch_size, verbose=-1) @@ -503,7 +850,7 @@ def _check_eval_results(metrics, metric_key, metric_list): # metric_list: 多个用来做评价的指标，来自Trainer的初始化 if isinstance(metrics, tuple): loss, metrics = metrics - + if isinstance(metrics, dict): if len(metrics) == 1: # only single metric, just use it @@ -514,7 +861,7 @@ def _check_eval_results(metrics, metric_key, metric_list): if metrics_name not in metrics: raise RuntimeError(f"{metrics_name} is chosen to do validation, but got {metrics}") metric_dict = metrics[metrics_name] - + if len(metric_dict) == 1: indicator_val, indicator = list(metric_dict.values())[0], list(metric_dict.keys())[0] elif len(metric_dict) > 1 and metric_key is None: diff --git a/fastNLP/core/utils.py b/fastNLP/core/utils.py index 2e0f383e..79af296b 100644 --- a/fastNLP/core/utils.py +++ b/fastNLP/core/utils.py @@ -1,59 +1,270 @@ +""" +utils模块实现了 fastNLP 内部和外部所需的很多工具。其中用户可以使用的是 :func:`cache_results` 修饰器。 +""" +__all__ = [ + "cache_results", + "seq_len_to_mask" +] + import _pickle import inspect import os import warnings -from collections import Counter -from collections import namedtuple +from collections import Counter, namedtuple import numpy as np import torch +import torch.nn as nn + -CheckRes = namedtuple('CheckRes', ['missing', 'unused', 'duplicated', 'required', 'all_needed', - 'varargs']) +_CheckRes = namedtuple('_CheckRes', ['missing', 'unused', 'duplicated', 'required', 'all_needed', + 'varargs']) -def save_pickle(obj, pickle_path, file_name): - """Save an object into a pickle file. +def _prepare_cache_filepath(filepath): + """ + 检查filepath是否可以作为合理的cache文件. 如果可以的话，会自动创造路径 + :param filepath: str. + :return: None, if not, this function will raise error + """ + _cache_filepath = os.path.abspath(filepath) + if os.path.isdir(_cache_filepath): + raise RuntimeError("The cache_file_path must be a file, not a directory.") + cache_dir = os.path.dirname(_cache_filepath) + if not os.path.exists(cache_dir): + os.makedirs(cache_dir) + - :param obj: an object - :param pickle_path: str, the directory where the pickle file is to be saved - :param file_name: str, the name of the pickle file. In general, it should be ended by "pkl". +# TODO 可以保存下缓存时的参数，如果load的时候发现参数不一致，发出警告。 +def cache_results(_cache_fp, _refresh=False, _verbose=1): + """ + 别名：:class:`fastNLP.cache_results` :class:`fastNLP.core.uitls.cache_results` + + cache_results是fastNLP中用于cache数据的装饰器。通过下面的例子看一下如何使用:: + + import time + import numpy as np + from fastNLP import cache_results + + @cache_results('cache.pkl') + def process_data(): + # 一些比较耗时的工作，比如读取数据，预处理数据等，这里用time.sleep()代替耗时 + time.sleep(1) + return np.random.randint(10, size=(5,)) + + start_time = time.time() + print("res =",process_data()) + print(time.time() - start_time) + + start_time = time.time() + print("res =",process_data()) + print(time.time() - start_time) + + # 输出内容如下，可以看到两次结果相同，且第二次几乎没有花费时间 + # Save cache to cache.pkl. + # res = [5 4 9 1 8] + # 1.0042750835418701 + # Read cache from cache.pkl. + # res = [5 4 9 1 8] + # 0.0040721893310546875 + + 可以看到第二次运行的时候，只用了0.0001s左右，是由于第二次运行将直接从cache.pkl这个文件读取数据，而不会经过再次预处理:: + + # 还是以上面的例子为例，如果需要重新生成另一个cache，比如另一个数据集的内容，通过如下的方式调用即可 + process_data(_cache_fp='cache2.pkl') # 完全不影响之前的‘cache.pkl' + + 上面的_cache_fp是cache_results会识别的参数，它将从'cache2.pkl'这里缓存/读取数据，即这里的'cache2.pkl'覆盖默认的 + 'cache.pkl'。如果在你的函数前面加上了@cache_results()则你的函数会增加三个参数[_cache_fp, _refresh, _verbose]。 + 上面的例子即为使用_cache_fp的情况，这三个参数不会传入到你的函数中，当然你写的函数参数名也不可能包含这三个名称:: + + process_data(_cache_fp='cache2.pkl', _refresh=True) # 这里强制重新生成一份对预处理的cache。 + # _verbose是用于控制输出信息的，如果为0,则不输出任何内容;如果为1,则会提醒当前步骤是读取的cache还是生成了新的cache + + :param str _cache_fp: 将返回结果缓存到什么位置;或从什么位置读取缓存。如果为None，cache_results没有任何效用，除非在 + 函数调用的时候传入_cache_fp这个参数。 + :param bool _refresh: 是否重新生成cache。 + :param int _verbose: 是否打印cache的信息。 + :return: """ - if not os.path.exists(pickle_path): - os.mkdir(pickle_path) - print("make dir {} before saving pickle file".format(pickle_path)) - with open(os.path.join(pickle_path, file_name), "wb") as f: - _pickle.dump(obj, f) - print("{} saved in {}".format(file_name, pickle_path)) + + def wrapper_(func): + signature = inspect.signature(func) + for key, _ in signature.parameters.items(): + if key in ('_cache_fp', '_refresh', '_verbose'): + raise RuntimeError("The function decorated by cache_results cannot have keyword `{}`.".format(key)) + + def wrapper(*args, **kwargs): + if '_cache_fp' in kwargs: + cache_filepath = kwargs.pop('_cache_fp') + assert isinstance(cache_filepath, str), "_cache_fp can only be str." + else: + cache_filepath = _cache_fp + if '_refresh' in kwargs: + refresh = kwargs.pop('_refresh') + assert isinstance(refresh, bool), "_refresh can only be bool." + else: + refresh = _refresh + if '_verbose' in kwargs: + verbose = kwargs.pop('_verbose') + assert isinstance(verbose, int), "_verbose can only be integer." + else: + verbose = _verbose + refresh_flag = True + + if cache_filepath is not None and refresh is False: + # load data + if os.path.exists(cache_filepath): + with open(cache_filepath, 'rb') as f: + results = _pickle.load(f) + if verbose == 1: + print("Read cache from {}.".format(cache_filepath)) + refresh_flag = False + + if refresh_flag: + results = func(*args, **kwargs) + if cache_filepath is not None: + if results is None: + raise RuntimeError("The return value is None. Delete the decorator.") + _prepare_cache_filepath(cache_filepath) + with open(cache_filepath, 'wb') as f: + _pickle.dump(results, f) + print("Save cache to {}.".format(cache_filepath)) + + return results + + return wrapper + + return wrapper_ + + +# def save_pickle(obj, pickle_path, file_name): +# """Save an object into a pickle file. +# +# :param obj: an object +# :param pickle_path: str, the directory where the pickle file is to be saved +# :param file_name: str, the name of the pickle file. In general, it should be ended by "pkl". +# """ +# if not os.path.exists(pickle_path): +# os.mkdir(pickle_path) +# print("make dir {} before saving pickle file".format(pickle_path)) +# with open(os.path.join(pickle_path, file_name), "wb") as f: +# _pickle.dump(obj, f) +# print("{} saved in {}".format(file_name, pickle_path)) +# +# +# def load_pickle(pickle_path, file_name): +# """Load an object from a given pickle file. +# +# :param pickle_path: str, the directory where the pickle file is. +# :param file_name: str, the name of the pickle file. +# :return obj: an object stored in the pickle +# """ +# with open(os.path.join(pickle_path, file_name), "rb") as f: +# obj = _pickle.load(f) +# print("{} loaded from {}".format(file_name, pickle_path)) +# return obj +# +# +# def pickle_exist(pickle_path, pickle_name): +# """Check if a given pickle file exists in the directory. +# +# :param pickle_path: the directory of target pickle file +# :param pickle_name: the filename of target pickle file +# :return: True if file exists else False +# """ +# if not os.path.exists(pickle_path): +# os.makedirs(pickle_path) +# file_name = os.path.join(pickle_path, pickle_name) +# if os.path.exists(file_name): +# return True +# else: +# return False + +def _move_model_to_device(model, device): + """ + 将model移动到device + + :param model: torch.nn.DataParallel or torch.nn.Module. 当为torch.nn.DataParallel, 则只是调用一次cuda。device必须为 + None。 + :param str,int,torch.device,list(int),list(torch.device) device: 将模型load到哪个设备。默认为None，即Trainer不对模型 + 的计算位置进行管理。支持以下的输入: + + 1. str: ['cpu', 'cuda', 'cuda:0', 'cuda:1', ...] 依次为'cpu'中, 可见的第一个GPU中, 可见的第一个GPU中, + 可见的第二个GPU中; + + 2. torch.device：将模型装载到torch.device上。 + 3. int: 将使用device_id为该值的gpu进行训练 -def load_pickle(pickle_path, file_name): - """Load an object from a given pickle file. + 4. list(int)：如果多于1个device，将使用torch.nn.DataParallel包裹model, 并使用传入的device。 - :param pickle_path: str, the directory where the pickle file is. - :param file_name: str, the name of the pickle file. - :return obj: an object stored in the pickle + 5. None. 为None则不对模型进行任何处理，如果传入的model为torch.nn.DataParallel该值必须为None。 + + :return: torch.nn.DataParallel or torch.nn.Module """ - with open(os.path.join(pickle_path, file_name), "rb") as f: - obj = _pickle.load(f) - print("{} loaded from {}".format(file_name, pickle_path)) - return obj + if isinstance(model, torch.nn.parallel.DistributedDataParallel): + raise RuntimeError("model of `torch.nn.parallel.DistributedDataParallel` is not supported right now.") + + if device is None: + if isinstance(model, torch.nn.DataParallel): + model.cuda() + return model + else: + if not torch.cuda.is_available() and ( + device != 'cpu' or (isinstance(device, torch.device) and device.type != 'cpu')): + raise ValueError("There is no usable gpu. set `device` as `cpu` or `None`.") + + if isinstance(model, torch.nn.DataParallel): + raise RuntimeError("When model is `torch.nn.DataParallel`, the device has to be `None`.") + + if isinstance(device, int): + assert device > -1, "device can only be non-negative integer" + assert torch.cuda.device_count() > device, "Only has {} gpus, cannot use device {}.".format( + torch.cuda.device_count(), + device) + device = torch.device('cuda:{}'.format(device)) + elif isinstance(device, str): + device = torch.device(device) + if device.type == 'cuda' and device.index is not None: + assert device.index < torch.cuda.device_count(), "Only has {} gpus, cannot use device cuda:{}.".format( + torch.cuda.device_count(), + device) + elif isinstance(device, torch.device): + if device.type == 'cuda' and device.index is not None: + assert device.index < torch.cuda.device_count(), "Only has {} gpus, cannot use device cuda:{}.".format( + torch.cuda.device_count(), + device) + elif isinstance(device, list): + types = set([type(d) for d in device]) + assert len(types) == 1, "Mixed type in device, only `int` allowed." + assert list(types)[0] == int, "Only int supported for multiple devices." + assert len(set(device)) == len(device), "Duplicated device id found in device." + for d in device: + assert d > -1, "Only non-negative device id allowed." + if len(device) > 1: + output_device = device[0] + model = nn.DataParallel(model, device_ids=device, output_device=output_device) + device = torch.device(device[0]) + else: + raise TypeError("Unsupported device type.") + model = model.to(device) + return model -def pickle_exist(pickle_path, pickle_name): - """Check if a given pickle file exists in the directory. +def _get_model_device(model): + """ + 传入一个nn.Module的模型，获取它所在的device - :param pickle_path: the directory of target pickle file - :param pickle_name: the filename of target pickle file - :return: True if file exists else False + :param model: nn.Module + :return: torch.device,None 如果返回值为None，说明这个模型没有任何参数。 """ - if not os.path.exists(pickle_path): - os.makedirs(pickle_path) - file_name = os.path.join(pickle_path, pickle_name) - if os.path.exists(file_name): - return True + assert isinstance(model, nn.Module) + + parameters = list(model.parameters()) + if len(parameters) == 0: + return None else: - return False + return parameters[0].device def _build_args(func, **kwargs): @@ -126,30 +337,35 @@ def _check_arg_dict_list(func, args): missing = list(require_args - input_args) unused = list(input_args - all_args) varargs = [] if not spect.varargs else [spect.varargs] - return CheckRes(missing=missing, - unused=unused, - duplicated=duplicated, - required=list(require_args), - all_needed=list(all_args), - varargs=varargs) + return _CheckRes(missing=missing, + unused=unused, + duplicated=duplicated, + required=list(require_args), + all_needed=list(all_args), + varargs=varargs) -def get_func_signature(func): +def _get_func_signature(func): """ Given a function or method, return its signature. For example: - (1) function + + 1 function:: + def func(a, b='a', *args): xxxx get_func_signature(func) # 'func(a, b='a', *args)' - (2) method + + 2 method:: + class Demo: def __init__(self): xxx def forward(self, a, b='a', **args) demo = Demo() get_func_signature(demo.forward) # 'Demo.forward(self, a, b='a', **args)' + :param func: a function or a method :return: str or None """ @@ -186,35 +402,39 @@ def _check_function_or_method(func): raise TypeError(f"{type(func)} is not a method or function.") -def _move_dict_value_to_device(*args, device: torch.device): +def _move_dict_value_to_device(*args, device: torch.device, non_blocking=False): """ move data to model's device, element in *args should be dict. This is a inplace change. :param device: torch.device + :param non_blocking: bool, 是否异步将数据转移到cpu, 需要tensor使用pin_memory() :param args: :return: """ + if not torch.cuda.is_available(): + return + if not isinstance(device, torch.device): raise TypeError(f"device must be `torch.device`, got `{type(device)}`") - + for arg in args: if isinstance(arg, dict): for key, value in arg.items(): if isinstance(value, torch.Tensor): - arg[key] = value.to(device) + arg[key] = value.to(device, non_blocking=non_blocking) else: raise TypeError("Only support `dict` type right now.") -class CheckError(Exception): +class _CheckError(Exception): """ - CheckError. Used in losses.LossBase, metrics.MetricBase. + _CheckError. Used in losses.LossBase, metrics.MetricBase. """ - - def __init__(self, check_res: CheckRes, func_signature: str): + + def __init__(self, check_res: _CheckRes, func_signature: str): errs = [f'Problems occurred when calling `{func_signature}`'] - + if check_res.varargs: errs.append(f"\tvarargs: {check_res.varargs}(Does not support pass positional arguments, please delete it)") if check_res.missing: @@ -223,9 +443,9 @@ class CheckError(Exception): errs.append(f"\tduplicated param: {check_res.duplicated}") if check_res.unused: errs.append(f"\tunused param: {check_res.unused}") - + Exception.__init__(self, '\n'.join(errs)) - + self.check_res = check_res self.func_signature = func_signature @@ -235,7 +455,7 @@ WARNING_CHECK_LEVEL = 1 STRICT_CHECK_LEVEL = 2 -def _check_loss_evaluate(prev_func_signature: str, func_signature: str, check_res: CheckRes, +def _check_loss_evaluate(prev_func_signature: str, func_signature: str, check_res: _CheckRes, pred_dict: dict, target_dict: dict, dataset, check_level=0): errs = [] unuseds = [] @@ -245,7 +465,7 @@ def _check_loss_evaluate(prev_func_signature: str, func_signature: str, check_re # if check_res.varargs: # errs.append(f"\tvarargs: *{check_res.varargs}") # suggestions.append(f"Does not support pass positional arguments, please delete *{check_res.varargs}.") - + if check_res.unused: for _unused in check_res.unused: if _unused in target_dict: @@ -255,20 +475,19 @@ def _check_loss_evaluate(prev_func_signature: str, func_signature: str, check_re if _unused_field: unuseds.append(f"\tunused field: {_unused_field}") if _unused_param: - unuseds.append(f"\tunused param: {_unused_param}") # output from predict or forward - + unuseds.append(f"\tunused param: {_unused_param}") # output from predict or forward + module_name = func_signature.split('.')[0] if check_res.missing: errs.append(f"\tmissing param: {check_res.missing}") import re - mapped_missing = [] - unmapped_missing = [] + mapped_missing = [] # 提供了映射的参数 + unmapped_missing = [] # 没有指定映射的参数 input_func_map = {} - for _miss in check_res.missing: - if '(' in _miss: - # if they are like 'SomeParam(assign to xxx)' - _miss = _miss.split('(')[0] - matches = re.findall("(?<=`)[a-zA-Z0-9]*?(?=`)", _miss) + for _miss_ in check_res.missing: + # they shoudl like 'SomeParam(assign to xxx)' + _miss = _miss_.split('(')[0] + matches = re.findall("(?<=`)[a-zA-Z0-9]*?(?=`)", _miss_) if len(matches) == 2: fun_arg, module_name = matches input_func_map[_miss] = fun_arg @@ -278,50 +497,50 @@ def _check_loss_evaluate(prev_func_signature: str, func_signature: str, check_re mapped_missing.append(_miss) else: unmapped_missing.append(_miss) - - for _miss in mapped_missing: + + for _miss in mapped_missing + unmapped_missing: if _miss in dataset: - suggestions.append(f"Set {_miss} as target.") + suggestions.append(f"Set `{_miss}` as target.") else: _tmp = '' if check_res.unused: - _tmp = f"Check key assignment for `{input_func_map.get(_miss, _miss)}` when initialize {module_name}." + _tmp = f"Check key assignment for `{input_func_map.get(_miss,_miss)}` when initialize {module_name}." if _tmp: - _tmp += f' Or provide {_miss} in DataSet or output of {prev_func_signature}.' + _tmp += f' Or provide `{_miss}` in DataSet or output of {prev_func_signature}.' else: - _tmp = f'Provide {_miss} in DataSet or output of {prev_func_signature}.' + _tmp = f'Provide `{_miss}` in DataSet or output of {prev_func_signature}.' suggestions.append(_tmp) - for _miss in unmapped_missing: - if _miss in dataset: - suggestions.append(f"Set {_miss} as target.") - else: - _tmp = '' - if check_res.unused: - _tmp = f"Specify your assignment for `{input_func_map.get(_miss, _miss)}` when initialize {module_name}." - if _tmp: - _tmp += f' Or provide {_miss} in DataSet or output of {prev_func_signature}.' - else: - _tmp = f'Provide {_miss} in output of {prev_func_signature} or DataSet.' - suggestions.append(_tmp) - + # for _miss in unmapped_missing: + # if _miss in dataset: + # suggestions.append(f"Set `{_miss}` as target.") + # else: + # _tmp = '' + # if check_res.unused: + # _tmp = f"Specify your assignment for `{input_func_map.get(_miss, _miss)}` when initialize {module_name}." + # if _tmp: + # _tmp += f' Or provide `{_miss}` in DataSet or output of {prev_func_signature}.' + # else: + # _tmp = f'Provide `{_miss}` in output of {prev_func_signature} or DataSet.' + # suggestions.append(_tmp) + if check_res.duplicated: errs.append(f"\tduplicated param: {check_res.duplicated}.") suggestions.append(f"Delete {check_res.duplicated} in the output of " f"{prev_func_signature} or do not set {check_res.duplicated} as targets. ") - - if len(errs)>0: + + if len(errs) > 0: errs.extend(unuseds) elif check_level == STRICT_CHECK_LEVEL: errs.extend(unuseds) - + if len(errs) > 0: errs.insert(0, f'Problems occurred when calling {func_signature}') sugg_str = "" if len(suggestions) > 1: for idx, sugg in enumerate(suggestions): - if idx>0: + if idx > 0: sugg_str += '\t\t\t' - sugg_str += f'({idx+1}). {sugg}\n' + sugg_str += f'({idx + 1}). {sugg}\n' sugg_str = sugg_str[:-1] else: sugg_str += suggestions[0] @@ -336,14 +555,15 @@ def _check_loss_evaluate(prev_func_signature: str, func_signature: str, check_re _unused_warn = f'{check_res.unused} is not used by {module_name}.' warnings.warn(message=_unused_warn) + def _check_forward_error(forward_func, batch_x, dataset, check_level): check_res = _check_arg_dict_list(forward_func, batch_x) - func_signature = get_func_signature(forward_func) - + func_signature = _get_func_signature(forward_func) + errs = [] suggestions = [] _unused = [] - + # if check_res.varargs: # errs.append(f"\tvarargs: {check_res.varargs}") # suggestions.append(f"Does not support pass positional arguments, please delete *{check_res.varargs}.") @@ -364,20 +584,20 @@ def _check_forward_error(forward_func, batch_x, dataset, check_level): # _tmp += f"Or you might find it in `unused field:`, you can use DataSet.rename_field() to " \ # f"rename the field in `unused field:`." suggestions.append(_tmp) - + if check_res.unused: _unused = [f"\tunused field: {check_res.unused}"] - if len(errs)>0: + if len(errs) > 0: errs.extend(_unused) elif check_level == STRICT_CHECK_LEVEL: errs.extend(_unused) - + if len(errs) > 0: errs.insert(0, f'Problems occurred when calling {func_signature}') sugg_str = "" if len(suggestions) > 1: for idx, sugg in enumerate(suggestions): - sugg_str += f'({idx+1}). {sugg}' + sugg_str += f'({idx + 1}). {sugg}' else: sugg_str += suggestions[0] err_str = '\n' + '\n'.join(errs) + '\n\tSuggestion: ' + sugg_str @@ -388,72 +608,66 @@ def _check_forward_error(forward_func, batch_x, dataset, check_level): warnings.warn(message=_unused_warn) -def seq_lens_to_masks(seq_lens, float=False): +def seq_len_to_mask(seq_len): """ - Convert seq_lens to masks. - :param seq_lens: list, np.ndarray, or torch.LongTensor, shape should all be (B,) - :param float: if True, the return masks is in float type, otherwise it is byte. - :return: list, np.ndarray or torch.Tensor, shape will be (B, max_length) + 将一个表示sequence length的一维数组转换为二维的mask，不包含的位置为0。 + 转变 1-d seq_len到2-d mask. + + Example:: + + >>> seq_len = torch.arange(2, 16) + >>> mask = seq_len_to_mask(seq_len) + >>> print(mask.size()) + torch.Size([14, 15]) + >>> seq_len = np.arange(2, 16) + >>> mask = seq_len_to_mask(seq_len) + >>> print(mask.shape) + (14, 15) + + :param np.ndarray,torch.LongTensor seq_len: shape将是(B,) + :return: np.ndarray or torch.Tensor, shape将是(B, max_length)。 元素类似为bool或torch.uint8 """ - if isinstance(seq_lens, np.ndarray): - assert len(np.shape(seq_lens)) == 1, f"seq_lens can only have one dimension, got {len(np.shape(seq_lens))}." - assert seq_lens.dtype in (int, np.int32, np.int64), f"seq_lens can only be integer, not {seq_lens.dtype}." - raise NotImplemented - elif isinstance(seq_lens, torch.Tensor): - assert len(seq_lens.size()) == 1, f"seq_lens can only have one dimension, got {len(seq_lens.size())==1}." - batch_size = seq_lens.size(0) - max_len = seq_lens.max() - indexes = torch.arange(max_len).view(1, -1).repeat(batch_size, 1).to(seq_lens.device) - masks = indexes.lt(seq_lens.unsqueeze(1)) - - if float: - masks = masks.float() - - return masks - elif isinstance(seq_lens, list): - raise NotImplemented + if isinstance(seq_len, np.ndarray): + assert len(np.shape(seq_len)) == 1, f"seq_len can only have one dimension, got {len(np.shape(seq_len))}." + max_len = int(seq_len.max()) + broad_cast_seq_len = np.tile(np.arange(max_len), (len(seq_len), 1)) + mask = broad_cast_seq_len < seq_len.reshape(-1, 1) + + elif isinstance(seq_len, torch.Tensor): + assert seq_len.dim() == 1, f"seq_len can only have one dimension, got {seq_len.dim() == 1}." + batch_size = seq_len.size(0) + max_len = seq_len.max().long() + broad_cast_seq_len = torch.arange(max_len).expand(batch_size, -1).to(seq_len) + mask = broad_cast_seq_len.lt(seq_len.unsqueeze(1)) else: - raise NotImplemented - - -def seq_mask(seq_len, max_len): - """Create sequence mask. - - :param seq_len: list or torch.Tensor, the lengths of sequences in a batch. - :param max_len: int, the maximum sequence length in a batch. - :return mask: torch.LongTensor, [batch_size, max_len] - - """ - if not isinstance(seq_len, torch.Tensor): - seq_len = torch.LongTensor(seq_len) - seq_len = seq_len.view(-1, 1).long() # [batch_size, 1] - seq_range = torch.arange(start=0, end=max_len, dtype=torch.long, device=seq_len.device).view(1, -1) # [1, max_len] - return torch.gt(seq_len, seq_range) # [batch_size, max_len] + raise TypeError("Only support 1-d numpy.ndarray or 1-d torch.Tensor.") + + return mask -class pseudo_tqdm: +class _pseudo_tqdm: """ 当无法引入tqdm，或者Trainer中设置use_tqdm为false的时候，用该方法打印数据 """ - + def __init__(self, **kwargs): pass - + def write(self, info): print(info) - + def set_postfix_str(self, info): print(info) - + def __getattr__(self, item): def pass_func(*args, **kwargs): pass - + return pass_func - + def __enter__(self): return self - + def __exit__(self, exc_type, exc_val, exc_tb): del self diff --git a/fastNLP/core/vocabulary.py b/fastNLP/core/vocabulary.py index 50a79d24..cbde9cba 100644 --- a/fastNLP/core/vocabulary.py +++ b/fastNLP/core/vocabulary.py @@ -1,24 +1,33 @@ +__all__ = [ + "Vocabulary" +] + +from functools import wraps from collections import Counter +from .dataset import DataSet + -def check_build_vocab(func): +def _check_build_vocab(func): """A decorator to make sure the indexing is built before used. """ - + + @wraps(func) # to solve missing docstring def _wrapper(self, *args, **kwargs): if self.word2idx is None or self.rebuild is True: self.build_vocab() return func(self, *args, **kwargs) - + return _wrapper -def check_build_status(func): +def _check_build_status(func): """A decorator to check whether the vocabulary updates after the last build. """ - + + @wraps(func) # to solve missing docstring def _wrapper(self, *args, **kwargs): if self.rebuild is False: self.rebuild = True @@ -27,27 +36,36 @@ def check_build_status(func): "Adding more words may cause unexpected behaviour of Vocabulary. ".format( self.max_size, func.__name__)) return func(self, *args, **kwargs) - + return _wrapper class Vocabulary(object): - """Use for word and index one to one mapping - - Example:: + """ + 别名：:class:`fastNLP.Vocabulary` :class:`fastNLP.core.vocabulary.Vocabulary` + + 用于构建, 存储和使用 `str` 到 `int` 的一一映射:: vocab = Vocabulary() word_list = "this is a word list".split() vocab.update(word_list) - vocab["word"] - vocab.to_word(5) - - :param int max_size: set the max number of words in Vocabulary. Default: None - :param int min_freq: set the min occur frequency of words in Vocabulary. Default: None - + vocab["word"] # str to int + vocab.to_word(5) # int to str + + :param int max_size: `Vocabulary` 的最大大小, 即能存储词的最大数量 + 若为 ``None`` , 则不限制大小. Default: ``None`` + :param int min_freq: 能被记录下的词在文本中的最小出现频率, 应大于或等于 1. + 若小于该频率, 词语将被视为 `unknown`. 若为 ``None`` , 所有文本中的词都被记录. Default: ``None`` + :param str optional padding: padding的字符. 如果设置为 ``None`` , + 则vocabulary中不考虑padding, 也不计入词表大小，为 ``None`` 的情况多在为label建立Vocabulary的情况. + Default: '' + :param str optional unknown: unknown的字符，所有未被记录的词在转为 `int` 时将被视为unknown. + 如果设置为 ``None`` ,则vocabulary中不考虑unknow, 也不计入词表大小. + 为 ``None`` 的情况多在为label建立Vocabulary的情况. + Default: '' """ - - def __init__(self, max_size=None, min_freq=None, unknown='', padding=''): + + def __init__(self, max_size=None, min_freq=None, padding='', unknown=''): self.max_size = max_size self.min_freq = min_freq self.word_count = Counter() @@ -56,51 +74,55 @@ class Vocabulary(object): self.word2idx = None self.idx2word = None self.rebuild = True - - @check_build_status + + @_check_build_status def update(self, word_lst): - """Add a list of words into the vocabulary. + """依次增加序列中词在词典中的出现频率 :param list word_lst: a list of strings """ self.word_count.update(word_lst) - - @check_build_status + + @_check_build_status def add(self, word): - """Add a single word into the vocabulary. + """ + 增加一个新词在词典中的出现频率 - :param str word: a word or token. + :param str word: 新词 """ self.word_count[word] += 1 - - @check_build_status + + @_check_build_status def add_word(self, word): - """Add a single word into the vocabulary. - - :param str word: a word or token. + """ + 增加一个新词在词典中的出现频率 + :param str word: 新词 """ self.add(word) - - @check_build_status + + @_check_build_status def add_word_lst(self, word_lst): - """Add a list of words into the vocabulary. - - :param list word_lst: a list of strings + """ + 依次增加序列中词在词典中的出现频率 + :param list[str] word_lst: 词的序列 """ self.update(word_lst) - + def build_vocab(self): - """Build a mapping from word to index, and filter the word using ``max_size`` and ``min_freq``. + """ + 根据已经出现的词和出现频率构建词典. 注意: 重复构建可能会改变词典的大小, + 但已经记录在词典中的词, 不会改变对应的 `int` """ - self.word2idx = {} + if self.word2idx is None: + self.word2idx = {} if self.padding is not None: - self.word2idx[self.padding] = 0 + self.word2idx[self.padding] = len(self.word2idx) if self.unknown is not None: - self.word2idx[self.unknown] = 1 - + self.word2idx[self.unknown] = len(self.word2idx) + max_size = min(self.max_size, len(self.word_count)) if self.max_size else None words = self.word_count.most_common(max_size) if self.min_freq is not None: @@ -111,32 +133,45 @@ class Vocabulary(object): self.word2idx.update({w: i + start_idx for i, (w, _) in enumerate(words)}) self.build_reverse_vocab() self.rebuild = False - + def build_reverse_vocab(self): - """Build "index to word" dict based on "word to index" dict. + """ + 基于 "word to index" dict, 构建 "index to word" dict. """ self.idx2word = {i: w for w, i in self.word2idx.items()} - - @check_build_vocab + + @_check_build_vocab def __len__(self): return len(self.word2idx) - - @check_build_vocab + + @_check_build_vocab def __contains__(self, item): - """Check if a word in vocabulary. + """ + 检查词是否被记录 :param item: the word :return: True or False """ return item in self.word2idx - + def has_word(self, w): - return self.__contains__(w) + """ + 检查词是否被记录:: - @check_build_vocab + has_abc = vocab.has_word('abc') + # equals to + has_abc = 'abc' in vocab + + :param item: the word + :return: ``True`` or ``False`` + """ + return self.__contains__(w) + + @_check_build_vocab def __getitem__(self, w): - """To support usage like:: + """ + To support usage like:: vocab[w] """ @@ -146,49 +181,168 @@ class Vocabulary(object): return self.word2idx[self.unknown] else: raise ValueError("word {} not in vocabulary".format(w)) + + @_check_build_vocab + def index_dataset(self, *datasets, field_name, new_field_name=None): + """ + 将DataSet中对应field的词转为数字，Example:: + # remember to use `field_name` + vocab.index_dataset(train_data, dev_data, test_data, field_name='words') + + :param datasets: 需要转index的 class:`~fastNLP.DataSet` , 支持一个或多个（list） + :param str field_name: 需要转index的field, 若有多个 DataSet, 每个DataSet都必须有此 field. + 目前仅支持 ``str`` , ``list(str)`` , ``list(list(str))`` + :param str new_field_name: 保存结果的field_name. 若为 ``None`` , 将覆盖原field. + Default: ``None`` + """ + + def index_instance(ins): + """ + 有几种情况, str, 1d-list, 2d-list + :param ins: + :return: + """ + field = ins[field_name] + if isinstance(field, str): + return self.to_index(field) + elif isinstance(field, list): + if not isinstance(field[0], list): + return [self.to_index(w) for w in field] + else: + if isinstance(field[0][0], list): + raise RuntimeError("Only support field with 2 dimensions.") + return [[self.to_index(c) for c in w] for w in field] + + if new_field_name is None: + new_field_name = field_name + for idx, dataset in enumerate(datasets): + if isinstance(dataset, DataSet): + try: + dataset.apply(index_instance, new_field_name=new_field_name) + except Exception as e: + print("When processing the `{}` dataset, the following error occurred.".format(idx)) + raise e + else: + raise RuntimeError("Only DataSet type is allowed.") + + def from_dataset(self, *datasets, field_name): + """ + 使用dataset的对应field中词构建词典:: + + # remember to use `field_name` + vocab.from_dataset(train_data1, train_data2, field_name='words') + + :param datasets: 需要转index的 class:`~fastNLP.DataSet` , 支持一个或多个（list） + :param field_name: 可为 ``str`` 或 ``list(str)`` . + 构建词典所使用的 field(s), 支持一个或多个field + 若有多个 DataSet, 每个DataSet都必须有这些field. + 目前仅支持的field结构: ``str`` , ``list(str)`` , ``list(list(str))`` + :return self: + """ + if isinstance(field_name, str): + field_name = [field_name] + elif not isinstance(field_name, list): + raise TypeError('invalid argument field_name: {}'.format(field_name)) + + def construct_vocab(ins): + for fn in field_name: + field = ins[fn] + if isinstance(field, str): + self.add_word(field) + elif isinstance(field, list): + if not isinstance(field[0], list): + self.add_word_lst(field) + else: + if isinstance(field[0][0], list): + raise RuntimeError("Only support field with 2 dimensions.") + [self.add_word_lst(w) for w in field] + + for idx, dataset in enumerate(datasets): + if isinstance(dataset, DataSet): + try: + dataset.apply(construct_vocab) + except Exception as e: + print("When processing the `{}` dataset, the following error occurred.".format(idx)) + raise e + else: + raise RuntimeError("Only DataSet type is allowed.") + return self + def to_index(self, w): - """ Turn a word to an index. If w is not in Vocabulary, return the unknown label. + """ + 将词转为数字. 若词不再词典中被记录, 将视为 unknown, 若 ``unknown=None`` , 将抛出 + ``ValueError``:: + + index = vocab.to_index('abc') + # equals to + index = vocab['abc'] :param str w: a word + :return int index: the number """ return self.__getitem__(w) - + @property - @check_build_vocab + @_check_build_vocab def unknown_idx(self): + """ + unknown 对应的数字. + """ if self.unknown is None: return None return self.word2idx[self.unknown] - + @property - @check_build_vocab + @_check_build_vocab def padding_idx(self): + """ + padding 对应的数字 + """ if self.padding is None: return None return self.word2idx[self.padding] - - @check_build_vocab + + @_check_build_vocab def to_word(self, idx): - """given a word's index, return the word itself + """ + 给定一个数字, 将其转为对应的词. :param int idx: the index - :return str word: the indexed word + :return str word: the word """ return self.idx2word[idx] + + def clear(self): + """ + 删除Vocabulary中的词表数据。相当于重新初始化一下。 + :return: + """ + self.word_count.clear() + self.word2idx = None + self.idx2word = None + self.rebuild = True + def __getstate__(self): """Use to prepare data for pickle. """ + len(self) # make sure vocab has been built state = self.__dict__.copy() # no need to pickle idx2word as it can be constructed from word2idx del state['idx2word'] return state - + def __setstate__(self, state): """Use to restore state from pickle. """ self.__dict__.update(state) self.build_reverse_vocab() + + def __repr__(self): + return "Vocabulary({}...)".format(list(self.word_count.keys())[:5]) + + def __iter__(self): + return iter(list(self.word_count.keys())) diff --git a/fastNLP/io/__init__.py b/fastNLP/io/__init__.py index e69de29b..c8d6a441 100644 --- a/fastNLP/io/__init__.py +++ b/fastNLP/io/__init__.py @@ -0,0 +1,31 @@ +""" +用于IO的模块, 具体包括: + +1. 用于读入 embedding 的 :doc:`EmbedLoader ` 类, + +2. 用于读入数据的 :doc:`DataSetLoader ` 类 + +3. 用于保存和载入模型的类, 参考 :doc:`/fastNLP.io.model_io` + +这些类的使用方法如下: +""" +__all__ = [ + 'EmbedLoader', + + 'DataSetLoader', + 'CSVLoader', + 'JsonLoader', + 'ConllLoader', + 'SNLILoader', + 'SSTLoader', + 'PeopleDailyCorpusLoader', + 'Conll2003Loader', + + 'ModelLoader', + 'ModelSaver', +] + +from .embed_loader import EmbedLoader +from .dataset_loader import DataSetLoader, CSVLoader, JsonLoader, ConllLoader, SNLILoader, SSTLoader, \ + PeopleDailyCorpusLoader, Conll2003Loader +from .model_io import ModelLoader, ModelSaver diff --git a/fastNLP/io/base_loader.py b/fastNLP/io/base_loader.py index ccfa1169..4ab1e2d0 100644 --- a/fastNLP/io/base_loader.py +++ b/fastNLP/io/base_loader.py @@ -1,28 +1,46 @@ +__all__ = [ + "BaseLoader" +] + import _pickle as pickle import os class BaseLoader(object): - """Base loader for all loaders. + """ + 各个 Loader 的基类，提供了 API 的参考。 """ + def __init__(self): super(BaseLoader, self).__init__() - + @staticmethod def load_lines(data_path): + """ + 按行读取，舍弃每行两侧空白字符，返回list of str + + :param data_path: 读取数据的路径 + """ with open(data_path, "r", encoding="utf=8") as f: text = f.readlines() return [line.strip() for line in text] - + @classmethod def load(cls, data_path): + """ + 先按行读取，去除一行两侧空白，再提取每行的字符。返回list of list of str + + :param data_path: + """ with open(data_path, "r", encoding="utf-8") as f: text = f.readlines() return [[word for word in sent.strip()] for sent in text] - + @classmethod def load_with_cache(cls, data_path, cache_path): + """缓存版的load + """ if os.path.isfile(cache_path) and os.path.getmtime(data_path) < os.path.getmtime(cache_path): with open(cache_path, 'rb') as f: return pickle.load(f) @@ -34,22 +52,23 @@ class BaseLoader(object): class DataLoaderRegister: - """Register for all data sets. - - """ _readers = {} - + @classmethod def set_reader(cls, reader_cls, read_fn_name): # def wrapper(reader_cls): if read_fn_name in cls._readers: - raise KeyError('duplicate reader: {} and {} for read_func: {}'.format(cls._readers[read_fn_name], reader_cls, read_fn_name)) + raise KeyError( + 'duplicate reader: {} and {} for read_func: {}'.format(cls._readers[read_fn_name], reader_cls, + read_fn_name)) if hasattr(reader_cls, 'load'): cls._readers[read_fn_name] = reader_cls().load return reader_cls - + @classmethod def get_reader(cls, read_fn_name): if read_fn_name in cls._readers: return cls._readers[read_fn_name] raise AttributeError('no read function: {}'.format(read_fn_name)) + + # TODO 这个类使用在何处？ diff --git a/fastNLP/io/config_io.py b/fastNLP/io/config_io.py index 8be59a35..4acdbb96 100644 --- a/fastNLP/io/config_io.py +++ b/fastNLP/io/config_io.py @@ -1,36 +1,52 @@ +""" +用于读入和处理和保存 config 文件 + .. todo:: + 这个模块中的类可能被抛弃？ +""" +__all__ = [ + "ConfigLoader", + "ConfigSection", + "ConfigSaver" +] + import configparser import json import os -from fastNLP.io.base_loader import BaseLoader +from .base_loader import BaseLoader class ConfigLoader(BaseLoader): - """Loader for configuration. + """ + 别名：:class:`fastNLP.io.ConfigLoader` :class:`fastNLP.io.config_io.ConfigLoader` - :param str data_path: path to the config + 读取配置文件的Loader - """ + :param str data_path: 配置文件的路径 + """ + def __init__(self, data_path=None): super(ConfigLoader, self).__init__() if data_path is not None: self.config = self.parse(super(ConfigLoader, self).load(data_path)) - + @staticmethod def parse(string): raise NotImplementedError - + @staticmethod def load_config(file_path, sections): - """Load section(s) of configuration into the ``sections`` provided. No returns. + """ + 把配置文件的section 存入提供的 ``sections`` 中 - :param str file_path: the path of config file - :param dict sections: the dict of ``{section_name(string): ConfigSection object}`` + :param str file_path: 配置文件的路径 + :param dict sections: 符合如下键值对组成的字典 `section_name(string)` : :class:`~fastNLP.io.ConfigSection` + Example:: test_args = ConfigSection() - ConfigLoader("config.cfg", "").load_config("./data_for_tests/config", {"POS_test": test_args}) + ConfigLoader("config.cfg").load_config("./data_for_tests/config", {"POS_test": test_args}) """ assert isinstance(sections, dict) @@ -66,13 +82,16 @@ class ConfigLoader(BaseLoader): class ConfigSection(object): - """ConfigSection is the data structure storing all key-value pairs in one section in a config file. - """ + 别名：:class:`fastNLP.io.ConfigSection` :class:`fastNLP.io.config_io.ConfigSection` + ConfigSection是一个存储了一个section中所有键值对的数据结构，推荐使用此类的实例来配合 :meth:`ConfigLoader.load_config` 使用 + + """ + def __init__(self): super(ConfigSection, self).__init__() - + def __getitem__(self, key): """ :param key: str, the name of the attribute @@ -85,7 +104,7 @@ class ConfigSection(object): if key in self.__dict__.keys(): return getattr(self, key) raise AttributeError("do NOT have attribute %s" % key) - + def __setitem__(self, key, value): """ :param key: str, the name of the attribute @@ -100,14 +119,14 @@ class ConfigSection(object): raise AttributeError("attr %s except %s but got %s" % (key, str(type(getattr(self, key))), str(type(value)))) setattr(self, key, value) - + def __contains__(self, item): """ :param item: The key of item. :return: True if the key in self.__dict__.keys() else False. """ return item in self.__dict__.keys() - + def __eq__(self, other): """Overwrite the == operator @@ -119,15 +138,15 @@ class ConfigSection(object): return False if getattr(self, k) != getattr(self, k): return False - + for k in other.__dict__.keys(): if k not in self.__dict__.keys(): return False if getattr(self, k) != getattr(self, k): return False - + return True - + def __ne__(self, other): """Overwrite the != operator @@ -135,25 +154,30 @@ class ConfigSection(object): :return: """ return not self.__eq__(other) - + @property def data(self): return self.__dict__ class ConfigSaver(object): - """ConfigSaver is used to save config file and solve related conflicts. + """ + 别名：:class:`fastNLP.io.ConfigSaver` :class:`fastNLP.io.config_io.ConfigSaver` + + ConfigSaver 是用来存储配置文件并解决相关冲突的类 - :param str file_path: path to the config file + :param str file_path: 配置文件的路径 """ + def __init__(self, file_path): self.file_path = file_path if not os.path.exists(self.file_path): raise FileNotFoundError("file {} NOT found!".__format__(self.file_path)) - + def _get_section(self, sect_name): - """This is the function to get the section with the section name. + """ + This is the function to get the section with the section name. :param sect_name: The name of section what wants to load. :return: The section. @@ -161,25 +185,26 @@ class ConfigSaver(object): sect = ConfigSection() ConfigLoader().load_config(self.file_path, {sect_name: sect}) return sect - + def _read_section(self): - """This is the function to read sections from the config file. + """ + This is the function to read sections from the config file. :return: sect_list, sect_key_list sect_list: A list of ConfigSection(). sect_key_list: A list of names in sect_list. """ sect_name = None - + sect_list = {} sect_key_list = [] - + single_section = {} single_section_key = [] - + with open(self.file_path, 'r') as f: lines = f.readlines() - + for line in lines: if line.startswith('[') and line.endswith(']\n'): if sect_name is None: @@ -191,33 +216,32 @@ class ConfigSaver(object): sect_key_list.append(sect_name) sect_name = line[1: -2] continue - + if line.startswith('#'): single_section[line] = '#' single_section_key.append(line) continue - + if line.startswith('\n'): single_section_key.append('\n') continue - + if '=' not in line: - # log = create_logger(__name__, './config_saver.log') - # log.error("can NOT load config file [%s]" % self.file_path) raise RuntimeError("can NOT load config file {}".__format__(self.file_path)) - + key = line.split('=', maxsplit=1)[0].strip() value = line.split('=', maxsplit=1)[1].strip() + '\n' single_section[key] = value single_section_key.append(key) - + if sect_name is not None: sect_list[sect_name] = single_section, single_section_key sect_key_list.append(sect_name) return sect_list, sect_key_list - + def _write_section(self, sect_list, sect_key_list): - """This is the function to write config file with section list and name list. + """ + This is the function to write config file with section list and name list. :param sect_list: A list of ConfigSection() need to be writen into file. :param sect_key_list: A list of name of sect_list. @@ -236,12 +260,13 @@ class ConfigSaver(object): continue f.write(key + ' = ' + single_section[key]) f.write('\n') - + def save_config_file(self, section_name, section): - """This is the function to be called to change the config file with a single section and its name. + """ + 这个方法可以用来修改并保存配置文件中单独的一个 section - :param str section_name: The name of section what needs to be changed and saved. - :param ConfigSection section: The section with key and value what needs to be changed and saved. + :param str section_name: 需要保存的 section 的名字. + :param section: 你需要修改并保存的 section， :class:`~fastNLP.io.ConfigSaver` 类型 """ section_file = self._get_section(section_name) if len(section_file.__dict__.keys()) == 0: # the section not in the file before @@ -263,19 +288,15 @@ class ConfigSaver(object): change_file = True break if section_file[k] != section[k]: - # logger = create_logger(__name__, "./config_loader.log") - # logger.warning("section [%s] in config file [%s] has been changed" % ( - # section_name, self.file_path - # )) change_file = True break if not change_file: return - + sect_list, sect_key_list = self._read_section() if section_name not in sect_key_list: raise AttributeError() - + sect, sect_key = sect_list[section_name] for k in section.__dict__.keys(): if k not in sect_key: diff --git a/fastNLP/io/dataset_loader.py b/fastNLP/io/dataset_loader.py index 27d8a360..0abaa42b 100644 --- a/fastNLP/io/dataset_loader.py +++ b/fastNLP/io/dataset_loader.py @@ -1,379 +1,198 @@ -import os - -from fastNLP.core.dataset import DataSet -from fastNLP.core.instance import Instance -from fastNLP.io.base_loader import DataLoaderRegister - - -def convert_seq_dataset(data): - """Create an DataSet instance that contains no labels. - - :param data: list of list of strings, [num_examples, *]. - Example:: - - [ - [word_11, word_12, ...], - ... - ] +""" +dataset_loader模块实现了许多 DataSetLoader, 用于读取不同格式的数据, 并返回 `DataSet` , +得到的 :class:`~fastNLP.DataSet` 对象可以直接传入 :class:`~fastNLP.Trainer` 和 :class:`~fastNLP.Tester`, 用于模型的训练和测试。 +以SNLI数据集为例:: - :return: a DataSet. - """ - dataset = DataSet() - for word_seq in data: - dataset.append(Instance(word_seq=word_seq)) - return dataset - - -def convert_seq2tag_dataset(data): - """Convert list of data into DataSet. + loader = SNLILoader() + train_ds = loader.load('path/to/train') + dev_ds = loader.load('path/to/dev') + test_ds = loader.load('path/to/test') - :param data: list of list of strings, [num_examples, *]. - Example:: + # ... do stuff + +为 fastNLP 提供 DataSetLoader 的开发者请参考 :class:`~fastNLP.io.DataSetLoader` 的介绍。 +""" +__all__ = [ + 'DataInfo', + 'DataSetLoader', + 'CSVLoader', + 'JsonLoader', + 'ConllLoader', + 'SNLILoader', + 'SSTLoader', + 'PeopleDailyCorpusLoader', + 'Conll2003Loader', +] + +from nltk.tree import Tree + +from ..core.dataset import DataSet +from ..core.instance import Instance +from .file_reader import _read_csv, _read_json, _read_conll +from typing import Union, Dict +import os - [ - [ [word_11, word_12, ...], label_1 ], - [ [word_21, word_22, ...], label_2 ], - ... - ] - :return: a DataSet. +def _download_from_url(url, path): + try: + from tqdm.auto import tqdm + except: + from ..core.utils import _pseudo_tqdm as tqdm + import requests + + """Download file""" + r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}, stream=True) + chunk_size = 16 * 1024 + total_size = int(r.headers.get('Content-length', 0)) + with open(path, "wb") as file, \ + tqdm(total=total_size, unit='B', unit_scale=1, desc=path.split('/')[-1]) as t: + for chunk in r.iter_content(chunk_size): + if chunk: + file.write(chunk) + t.update(len(chunk)) + return + + +def _uncompress(src, dst): + import zipfile + import gzip + import tarfile + import os + + def unzip(src, dst): + with zipfile.ZipFile(src, 'r') as f: + f.extractall(dst) + + def ungz(src, dst): + with gzip.open(src, 'rb') as f, open(dst, 'wb') as uf: + length = 16 * 1024 # 16KB + buf = f.read(length) + while buf: + uf.write(buf) + buf = f.read(length) + + def untar(src, dst): + with tarfile.open(src, 'r:gz') as f: + f.extractall(dst) + + fn, ext = os.path.splitext(src) + _, ext_2 = os.path.splitext(fn) + if ext == '.zip': + unzip(src, dst) + elif ext == '.gz' and ext_2 != '.tar': + ungz(src, dst) + elif (ext == '.gz' and ext_2 == '.tar') or ext_2 == '.tgz': + untar(src, dst) + else: + raise ValueError('unsupported file {}'.format(src)) + + +class DataInfo: """ - dataset = DataSet() - for sample in data: - dataset.append(Instance(word_seq=sample[0], label=sample[1])) - return dataset - - -def convert_seq2seq_dataset(data): - """Convert list of data into DataSet. + 经过处理的数据信息，包括一系列数据集（比如：分开的训练集、验证集和测试集）及它们所用的词表和词嵌入。 - :param data: list of list of strings, [num_examples, *]. - Example:: - - [ - [ [word_11, word_12, ...], [label_1, label_1, ...] ], - [ [word_21, word_22, ...], [label_2, label_1, ...] ], - ... - ] - - :return: a DataSet. + :param vocabs: 从名称(字符串)到 :class:`~fastNLP.Vocabulary` 类型的dict + :param embeddings: 从名称(字符串)到一系列 embedding 的dict，参考 :class:`~fastNLP.io.EmbedLoader` + :param datasets: 从名称(字符串)到 :class:`~fastNLP.DataSet` 类型的dict """ - dataset = DataSet() - for sample in data: - dataset.append(Instance(word_seq=sample[0], label_seq=sample[1])) - return dataset + + def __init__(self, vocabs: dict = None, embeddings: dict = None, datasets: dict = None): + self.vocabs = vocabs or {} + self.embeddings = embeddings or {} + self.datasets = datasets or {} class DataSetLoader: - """Interface for all DataSetLoaders. - - """ - - def load(self, path): - """Load data from a given file. - - :param str path: file path - :return: a DataSet object - """ - raise NotImplementedError - - def convert(self, data): - """Optional operation to build a DataSet. - - :param data: inner data structure (user-defined) to represent the data. - :return: a DataSet object - """ - raise NotImplementedError - - -class NativeDataSetLoader(DataSetLoader): - """A simple example of DataSetLoader - - """ - def __init__(self): - super(NativeDataSetLoader, self).__init__() - - def load(self, path): - ds = DataSet.read_csv(path, headers=("raw_sentence", "label"), sep="\t") - ds.set_input("raw_sentence") - ds.set_target("label") - return ds - - -DataLoaderRegister.set_reader(NativeDataSetLoader, 'read_naive') - - -class RawDataSetLoader(DataSetLoader): - """A simple example of raw data reader - """ - def __init__(self): - super(RawDataSetLoader, self).__init__() + 别名：:class:`fastNLP.io.DataSetLoader` :class:`fastNLP.io.dataset_loader.DataSetLoader` - def load(self, data_path, split=None): - with open(data_path, "r", encoding="utf-8") as f: - lines = f.readlines() - lines = lines if split is None else [l.split(split) for l in lines] - lines = list(filter(lambda x: len(x) > 0, lines)) - return self.convert(lines) - - def convert(self, data): - return convert_seq_dataset(data) - - -DataLoaderRegister.set_reader(RawDataSetLoader, 'read_rawdata') - - -class POSDataSetLoader(DataSetLoader): - """Dataset Loader for a POS Tag dataset. - - In these datasets, each line are divided by "\t". The first Col is the vocabulary and the second - Col is the label. Different sentence are divided by an empty line. - E.g:: - - Tom label1 - and label2 - Jerry label1 - . label3 - (separated by an empty line) - Hello label4 - world label5 - ! label3 - - In this example, there are two sentences "Tom and Jerry ." and "Hello world !". Each word has its own label. + 定义了各种 DataSetLoader (针对特定数据上的特定任务) 所需的API 接口，开发者应该继承它实现各种的 DataSetLoader。 + + 开发者至少应该编写如下内容: + + - _load 函数：从一个数据文件中读取数据到一个 :class:`~fastNLP.DataSet` + - load 函数（可以使用基类的方法）：从一个或多个数据文件中读取数据到一个或多个 :class:`~fastNLP.DataSet` + - process 函数：一个或多个从数据文件中读取数据，并处理成可以训练的一个或多个 :class:`~fastNLP.DataSet` + + **process 函数中可以 调用load 函数或 _load 函数** + """ - def __init__(self): - super(POSDataSetLoader, self).__init__() - - def load(self, data_path): - """ - :return data: three-level list - Example:: - [ - [ [word_11, word_12, ...], [label_1, label_1, ...] ], - [ [word_21, word_22, ...], [label_2, label_1, ...] ], - ... - ] + + def _download(self, url: str, path: str, uncompress=True) -> str: """ - with open(data_path, "r", encoding="utf-8") as f: - lines = f.readlines() - data = self.parse(lines) - return self.convert(data) - - @staticmethod - def parse(lines): - data = [] - sentence = [] - for line in lines: - line = line.strip() - if len(line) > 1: - sentence.append(line.split('\t')) - else: - words = [] - labels = [] - for tokens in sentence: - words.append(tokens[0]) - labels.append(tokens[1]) - data.append([words, labels]) - sentence = [] - if len(sentence) != 0: - words = [] - labels = [] - for tokens in sentence: - words.append(tokens[0]) - labels.append(tokens[1]) - data.append([words, labels]) - return data + + 从 ``url`` 下载数据到 ``path``， 如果 ``uncompress`` 为 ``True`` ，自动解压。 - def convert(self, data): - """Convert lists of strings into Instances with Fields. + :param url: 下载的网站 + :param path: 下载到的目录 + :param uncompress: 是否自动解压缩 + :return: 数据的存放路径 """ - return convert_seq2seq_dataset(data) - - -DataLoaderRegister.set_reader(POSDataSetLoader, 'read_pos') - - -class TokenizeDataSetLoader(DataSetLoader): - """ - Data set loader for tokenization data sets - """ - - def __init__(self): - super(TokenizeDataSetLoader, self).__init__() - - def load(self, data_path, max_seq_len=32): - """Load pku dataset for Chinese word segmentation. - CWS (Chinese Word Segmentation) pku training dataset format: - 1. Each line is a sentence. - 2. Each word in a sentence is separated by space. - This function convert the pku dataset into three-level lists with labels . - B: beginning of a word - M: middle of a word - E: ending of a word - S: single character - - :param str data_path: path to the data set. - :param max_seq_len: int, the maximum length of a sequence. If a sequence is longer than it, split it into - several sequences. - :return: three-level lists + pdir = os.path.dirname(path) + os.makedirs(pdir, exist_ok=True) + _download_from_url(url, path) + if uncompress: + dst = os.path.join(pdir, 'data') + _uncompress(path, dst) + return dst + return path + + def load(self, paths: Union[str, Dict[str, str]]) -> Union[DataSet, Dict[str, DataSet]]: """ - assert isinstance(max_seq_len, int) and max_seq_len > 0 - with open(data_path, "r", encoding="utf-8") as f: - sentences = f.readlines() - data = [] - for sent in sentences: - tokens = sent.strip().split() - words = [] - labels = [] - for token in tokens: - if len(token) == 1: - words.append(token) - labels.append("S") - else: - words.append(token[0]) - labels.append("B") - for idx in range(1, len(token) - 1): - words.append(token[idx]) - labels.append("M") - words.append(token[-1]) - labels.append("E") - num_samples = len(words) // max_seq_len - if len(words) % max_seq_len != 0: - num_samples += 1 - for sample_idx in range(num_samples): - start = sample_idx * max_seq_len - end = (sample_idx + 1) * max_seq_len - seq_words = words[start:end] - seq_labels = labels[start:end] - data.append([seq_words, seq_labels]) - return self.convert(data) - - def convert(self, data): - return convert_seq2seq_dataset(data) + 从指定一个或多个路径中的文件中读取数据，返回一个或多个数据集 :class:`~fastNLP.DataSet` 。 + 如果处理多个路径，传入的 dict 中的 key 与返回的 dict 中的 key 保存一致。 - -class ClassDataSetLoader(DataSetLoader): - """Loader for a dummy classification data set""" - - def __init__(self): - super(ClassDataSetLoader, self).__init__() - - def load(self, data_path): - assert os.path.exists(data_path) - with open(data_path, "r", encoding="utf-8") as f: - lines = f.readlines() - data = self.parse(lines) - return self.convert(data) - - @staticmethod - def parse(lines): + :param Union[str, Dict[str, str]] paths: 文件路径 + :return: :class:`~fastNLP.DataSet` 类的对象或存储多个 :class:`~fastNLP.DataSet` 的字典 """ + if isinstance(paths, str): + return self._load(paths) + return {name: self._load(path) for name, path in paths.items()} + + def _load(self, path: str) -> DataSet: + """从指定路径的文件中读取数据,返回 :class:`~fastNLP.DataSet` 类型的对象 - :param lines: lines from dataset - :return: list(list(list())): the three level of lists are words, sentence, and dataset + :param str path: 文件路径 + :return: 一个 :class:`~fastNLP.DataSet` 类型的对象 """ - dataset = list() - for line in lines: - line = line.strip().split() - label = line[0] - words = line[1:] - if len(words) <= 1: - continue - - sentence = [words, label] - dataset.append(sentence) - return dataset - - def convert(self, data): - return convert_seq2tag_dataset(data) - - -class ConllLoader(DataSetLoader): - """loader for conll format files""" - - def __init__(self): - super(ConllLoader, self).__init__() - - def load(self, data_path): - with open(data_path, "r", encoding="utf-8") as f: - lines = f.readlines() - data = self.parse(lines) - return self.convert(data) - - @staticmethod - def parse(lines): + raise NotImplementedError + + def process(self, paths: Union[str, Dict[str, str]], **options) -> DataInfo: """ - :param list lines: a list containing all lines in a conll file. - :return: a 3D list + 对于特定的任务和数据集，读取并处理数据，返回处理DataInfo类对象或字典。 + + 从指定一个或多个路径中的文件中读取数据，DataInfo对象中可以包含一个或多个数据集 。 + 如果处理多个路径，传入的 dict 的 key 与返回DataInfo中的 dict 中的 key 保存一致。 + + 返回的 :class:`DataInfo` 对象有如下属性： + + - vocabs: 由从数据集中获取的词表组成的字典，每个词表 + - embeddings: (可选) 数据集对应的词嵌入 + - datasets: 一个dict，包含一系列 :class:`~fastNLP.DataSet` 类型的对象。其中 field 的命名参考 :mod:`~fastNLP.core.const` + + :param paths: 原始数据读取的路径 + :param options: 根据不同的任务和数据集，设计自己的参数 + :return: 返回一个 DataInfo """ - sentences = list() - tokens = list() - for line in lines: - if line[0] == "#": - # skip the comments - continue - if line == "\n": - sentences.append(tokens) - tokens = [] - continue - tokens.append(line.split()) - return sentences - - def convert(self, data): - pass - - -class LMDataSetLoader(DataSetLoader): - """Language Model Dataset Loader - - This loader produces data for language model training in a supervised way. - That means it has X and Y. - - """ - - def __init__(self): - super(LMDataSetLoader, self).__init__() - - def load(self, data_path): - if not os.path.exists(data_path): - raise FileNotFoundError("file {} not found.".format(data_path)) - with open(data_path, "r", encoding="utf=8") as f: - text = " ".join(f.readlines()) - tokens = text.strip().split() - data = self.sentence_cut(tokens) - return self.convert(data) - - def sentence_cut(self, tokens, sentence_length=15): - start_idx = 0 - data_set = [] - for idx in range(len(tokens) // sentence_length): - x = tokens[start_idx * idx: start_idx * idx + sentence_length] - y = tokens[start_idx * idx + 1: start_idx * idx + sentence_length + 1] - if start_idx * idx + sentence_length + 1 >= len(tokens): - # ad hoc - y.extend([""]) - data_set.append([x, y]) - return data_set - - def convert(self, data): - pass + raise NotImplementedError class PeopleDailyCorpusLoader(DataSetLoader): """ - People Daily Corpus: Chinese word segmentation, POS tag, NER - """ + 别名：:class:`fastNLP.io.PeopleDailyCorpusLoader` :class:`fastNLP.io.dataset_loader.PeopleDailyCorpusLoader` - def __init__(self): + 读取人民日报数据集 + """ + + def __init__(self, pos=True, ner=True): super(PeopleDailyCorpusLoader, self).__init__() - - def load(self, data_path): + self.pos = pos + self.ner = ner + + def _load(self, data_path): with open(data_path, "r", encoding="utf-8") as f: sents = f.readlines() - - pos_tag_examples = [] - ner_examples = [] + examples = [] for sent in sents: if len(sent) <= 2: continue @@ -407,136 +226,299 @@ class PeopleDailyCorpusLoader(DataSetLoader): sent_ner.append(ner_tag) sent_pos_tag.append(pos) sent_words.append(token) - pos_tag_examples.append([sent_words, sent_pos_tag]) - ner_examples.append([sent_words, sent_ner]) - # List[List[List[str], List[str]]] - # ner_examples not used - return self.convert(pos_tag_examples) - + example = [sent_words] + if self.pos is True: + example.append(sent_pos_tag) + if self.ner is True: + example.append(sent_ner) + examples.append(example) + return self.convert(examples) + def convert(self, data): + """ + + :param data: python 内置对象 + :return: 一个 :class:`~fastNLP.DataSet` 类型的对象 + """ data_set = DataSet() for item in data: - sent_words, sent_pos_tag = item[0], item[1] - data_set.append(Instance(words=sent_words, tags=sent_pos_tag)) - data_set.apply(lambda ins: len(ins), new_field_name="seq_len") - data_set.set_target("tags") - data_set.set_input("sent_words") - data_set.set_input("seq_len") + sent_words = item[0] + if self.pos is True and self.ner is True: + instance = Instance( + words=sent_words, pos_tags=item[1], ner=item[2]) + elif self.pos is True: + instance = Instance(words=sent_words, pos_tags=item[1]) + elif self.ner is True: + instance = Instance(words=sent_words, ner=item[1]) + else: + instance = Instance(words=sent_words) + data_set.append(instance) + data_set.apply(lambda ins: len(ins["words"]), new_field_name="seq_len") return data_set -class Conll2003Loader(DataSetLoader): - """Self-defined loader of conll2003 dataset +class ConllLoader(DataSetLoader): + """ + 别名：:class:`fastNLP.io.ConllLoader` :class:`fastNLP.io.dataset_loader.ConllLoader` + + 读取Conll格式的数据. 数据格式详见 http://conll.cemantix.org/2012/data.html + + 列号从0开始, 每列对应内容为:: + + Column Type + 0 Document ID + 1 Part number + 2 Word number + 3 Word itself + 4 Part-of-Speech + 5 Parse bit + 6 Predicate lemma + 7 Predicate Frameset ID + 8 Word sense + 9 Speaker/Author + 10 Named Entities + 11:N Predicate Arguments + N Coreference + + :param headers: 每一列数据的名称，需为List or Tuple of str。``header`` 与 ``indexes`` 一一对应 + :param indexes: 需要保留的数据列下标，从0开始。若为 ``None`` ，则所有列都保留。Default: ``None`` + :param dropna: 是否忽略非法数据，若 ``False`` ，遇到非法数据时抛出 ``ValueError`` 。Default: ``False`` + """ - More information about the given dataset cound be found on - https://sites.google.com/site/ermasoftware/getting-started/ne-tagging-conll2003-data + def __init__(self, headers, indexes=None, dropna=False): + super(ConllLoader, self).__init__() + if not isinstance(headers, (list, tuple)): + raise TypeError( + 'invalid headers: {}, should be list of strings'.format(headers)) + self.headers = headers + self.dropna = dropna + if indexes is None: + self.indexes = list(range(len(self.headers))) + else: + if len(indexes) != len(headers): + raise ValueError + self.indexes = indexes + def _load(self, path): + ds = DataSet() + for idx, data in _read_conll(path, indexes=self.indexes, dropna=self.dropna): + ins = {h: data[i] for i, h in enumerate(self.headers)} + ds.append(Instance(**ins)) + return ds + + +class Conll2003Loader(ConllLoader): """ + 别名：:class:`fastNLP.io.Conll2003Loader` :class:`fastNLP.io.dataset_loader.Conll2003Loader` + + 读取Conll2003数据 + 关于数据集的更多信息,参考: + https://sites.google.com/site/ermasoftware/getting-started/ne-tagging-conll2003-data + """ + def __init__(self): - super(Conll2003Loader, self).__init__() - - def load(self, dataset_path): - with open(dataset_path, "r", encoding="utf-8") as f: - lines = f.readlines() - - ##Parse the dataset line by line - parsed_data = [] - sentence = [] - tokens = [] - for line in lines: - if '-DOCSTART- -X- -X- O' in line or line == '\n': - if sentence != []: - parsed_data.append((sentence, tokens)) - sentence = [] - tokens = [] - continue + headers = [ + 'tokens', 'pos', 'chunks', 'ner', + ] + super(Conll2003Loader, self).__init__(headers=headers) - temp = line.strip().split(" ") - sentence.append(temp[0]) - tokens.append(temp[1:4]) - return self.convert(parsed_data) +def _cut_long_sentence(sent, max_sample_length=200): + """ + 将长于max_sample_length的sentence截成多段，只会在有空格的地方发生截断。 + 所以截取的句子可能长于或者短于max_sample_length - def convert(self, parsed_data): - dataset = DataSet() - for sample in parsed_data: - label0_list = list(map( - lambda labels: labels[0], sample[1])) - label1_list = list(map( - lambda labels: labels[1], sample[1])) - label2_list = list(map( - lambda labels: labels[2], sample[1])) - dataset.append(Instance(token_list=sample[0], - label0_list=label0_list, - label1_list=label1_list, - label2_list=label2_list)) + :param sent: str. + :param max_sample_length: int. + :return: list of str. + """ + sent_no_space = sent.replace(' ', '') + cutted_sentence = [] + if len(sent_no_space) > max_sample_length: + parts = sent.strip().split() + new_line = '' + length = 0 + for part in parts: + length += len(part) + new_line += part + ' ' + if length > max_sample_length: + new_line = new_line[:-1] + cutted_sentence.append(new_line) + length = 0 + new_line = '' + if new_line != '': + cutted_sentence.append(new_line[:-1]) + else: + cutted_sentence.append(sent) + return cutted_sentence + + +class SSTLoader(DataSetLoader): + """ + 别名：:class:`fastNLP.io.SSTLoader` :class:`fastNLP.io.dataset_loader.SSTLoader` - return dataset + 读取SST数据集, DataSet包含fields:: + words: list(str) 需要分类的文本 + target: str 文本的标签 -class SNLIDataSetLoader(DataSetLoader): - """A data set loader for SNLI data set. + 数据来源: https://nlp.stanford.edu/sentiment/trainDevTestTrees_PTB.zip + :param subtree: 是否将数据展开为子树，扩充数据量. Default: ``False`` + :param fine_grained: 是否使用SST-5标准，若 ``False`` , 使用SST-2。Default: ``False`` """ - - def __init__(self): - super(SNLIDataSetLoader, self).__init__() - - def load(self, path_list): + + def __init__(self, subtree=False, fine_grained=False): + self.subtree = subtree + + tag_v = {'0': 'very negative', '1': 'negative', '2': 'neutral', + '3': 'positive', '4': 'very positive'} + if not fine_grained: + tag_v['0'] = tag_v['1'] + tag_v['4'] = tag_v['3'] + self.tag_v = tag_v + + def _load(self, path): """ - :param list path_list: A list of file name, in the order of premise file, hypothesis file, and label file. - :return: A DataSet object. + :param str path: 存储数据的路径 + :return: 一个 :class:`~fastNLP.DataSet` 类型的对象 """ - assert len(path_list) == 3 - line_set = [] - for file in path_list: - if not os.path.exists(file): - raise FileNotFoundError("file {} NOT found".format(file)) + datalist = [] + with open(path, 'r', encoding='utf-8') as f: + datas = [] + for l in f: + datas.extend([(s, self.tag_v[t]) + for s, t in self._get_one(l, self.subtree)]) + ds = DataSet() + for words, tag in datas: + ds.append(Instance(words=words, target=tag)) + return ds + + @staticmethod + def _get_one(data, subtree): + tree = Tree.fromstring(data) + if subtree: + return [(t.leaves(), t.label()) for t in tree.subtrees()] + return [(tree.leaves(), tree.label())] - with open(file, 'r', encoding='utf-8') as f: - lines = f.readlines() - line_set.append(lines) - premise_lines, hypothesis_lines, label_lines = line_set - assert len(premise_lines) == len(hypothesis_lines) and len(premise_lines) == len(label_lines) +class JsonLoader(DataSetLoader): + """ + 别名：:class:`fastNLP.io.JsonLoader` :class:`fastNLP.io.dataset_loader.JsonLoader` - data_set = [] - for premise, hypothesis, label in zip(premise_lines, hypothesis_lines, label_lines): - p = premise.strip().split() - h = hypothesis.strip().split() - l = label.strip() - data_set.append([p, h, l]) + 读取json格式数据.数据必须按行存储,每行是一个包含各类属性的json对象 - return self.convert(data_set) + :param dict fields: 需要读入的json属性名称, 和读入后在DataSet中存储的field_name + ``fields`` 的 `key` 必须是json对象的属性名. ``fields`` 的 `value` 为读入后在DataSet存储的 `field_name` , + `value` 也可为 ``None`` , 这时读入后的 `field_name` 与json对象对应属性同名 + ``fields`` 可为 ``None`` , 这时,json对象所有属性都保存在DataSet中. Default: ``None`` + :param bool dropna: 是否忽略非法数据,若 ``True`` 则忽略,若 ``False`` ,在遇到非法数据时,抛出 ``ValueError`` . + Default: ``False`` + """ + + def __init__(self, fields=None, dropna=False): + super(JsonLoader, self).__init__() + self.dropna = dropna + self.fields = None + self.fields_list = None + if fields: + self.fields = {} + for k, v in fields.items(): + self.fields[k] = k if v is None else v + self.fields_list = list(self.fields.keys()) + + def _load(self, path): + ds = DataSet() + for idx, d in _read_json(path, fields=self.fields_list, dropna=self.dropna): + if self.fields: + ins = {self.fields[k]: v for k, v in d.items()} + else: + ins = d + ds.append(Instance(**ins)) + return ds - def convert(self, data): - """Convert a 3D list to a DataSet object. - :param data: A 3D tensor. - Example:: - [ - [ [premise_word_11, premise_word_12, ...], [hypothesis_word_11, hypothesis_word_12, ...], [label_1] ], - [ [premise_word_21, premise_word_22, ...], [hypothesis_word_21, hypothesis_word_22, ...], [label_2] ], - ... - ] +class SNLILoader(JsonLoader): + """ + 别名：:class:`fastNLP.io.SNLILoader` :class:`fastNLP.io.dataset_loader.SNLILoader` - :return: A DataSet object. - """ + 读取SNLI数据集，读取的DataSet包含fields:: - data_set = DataSet() + words1: list(str)，第一句文本, premise + words2: list(str), 第二句文本, hypothesis + target: str, 真实标签 - for example in data: - p, h, l = example - # list, list, str - instance = Instance() - instance.add_field("premise", p) - instance.add_field("hypothesis", h) - instance.add_field("truth", l) - data_set.append(instance) - data_set.apply(lambda ins: len(ins["premise"]), new_field_name="premise_len") - data_set.apply(lambda ins: len(ins["hypothesis"]), new_field_name="hypothesis_len") - data_set.set_input("premise", "hypothesis", "premise_len", "hypothesis_len") - data_set.set_target("truth") - return data_set + 数据来源: https://nlp.stanford.edu/projects/snli/snli_1.0.zip + """ + + def __init__(self): + fields = { + 'sentence1_parse': 'words1', + 'sentence2_parse': 'words2', + 'gold_label': 'target', + } + super(SNLILoader, self).__init__(fields=fields) + + def _load(self, path): + ds = super(SNLILoader, self)._load(path) + + def parse_tree(x): + t = Tree.fromstring(x) + return t.leaves() + + ds.apply(lambda ins: parse_tree( + ins['words1']), new_field_name='words1') + ds.apply(lambda ins: parse_tree( + ins['words2']), new_field_name='words2') + ds.drop(lambda x: x['target'] == '-') + return ds + + +class CSVLoader(DataSetLoader): + """ + 别名：:class:`fastNLP.io.CSVLoader` :class:`fastNLP.io.dataset_loader.CSVLoader` + + 读取CSV格式的数据集。返回 ``DataSet`` + + :param List[str] headers: CSV文件的文件头.定义每一列的属性名称,即返回的DataSet中`field`的名称 + 若为 ``None`` ,则将读入文件的第一行视作 ``headers`` . Default: ``None`` + :param str sep: CSV文件中列与列之间的分隔符. Default: "," + :param bool dropna: 是否忽略非法数据,若 ``True`` 则忽略,若 ``False`` ,在遇到非法数据时,抛出 ``ValueError`` . + Default: ``False`` + """ + + def __init__(self, headers=None, sep=",", dropna=False): + self.headers = headers + self.sep = sep + self.dropna = dropna + + def _load(self, path): + ds = DataSet() + for idx, data in _read_csv(path, headers=self.headers, + sep=self.sep, dropna=self.dropna): + ds.append(Instance(**data)) + return ds + + +def _add_seg_tag(data): + """ + + :param data: list of ([word], [pos], [heads], [head_tags]) + :return: list of ([word], [pos]) + """ + + _processed = [] + for word_list, pos_list, _, _ in data: + new_sample = [] + for word, pos in zip(word_list, pos_list): + if len(word) == 1: + new_sample.append((word, 'S-' + pos)) + else: + new_sample.append((word[0], 'B-' + pos)) + for c in word[1:-1]: + new_sample.append((c, 'M-' + pos)) + new_sample.append((word[-1], 'E-' + pos)) + _processed.append(list(map(list, zip(*new_sample)))) + return _processed diff --git a/fastNLP/io/embed_loader.py b/fastNLP/io/embed_loader.py index e55fc55b..fb024e73 100644 --- a/fastNLP/io/embed_loader.py +++ b/fastNLP/io/embed_loader.py @@ -1,123 +1,155 @@ +__all__ = [ + "EmbedLoader" +] + +import os +import warnings + import numpy as np -import torch -from fastNLP.core.vocabulary import Vocabulary -from fastNLP.io.base_loader import BaseLoader +from ..core.vocabulary import Vocabulary +from .base_loader import BaseLoader class EmbedLoader(BaseLoader): - """docstring for EmbedLoader""" + """ + 别名：:class:`fastNLP.io.EmbedLoader` :class:`fastNLP.io.embed_loader.EmbedLoader` + 用于读取预训练的embedding, 读取结果可直接载入为模型参数。 + """ + def __init__(self): super(EmbedLoader, self).__init__() - + @staticmethod - def _load_glove(emb_file): - """Read file as a glove embedding - - file format: - embeddings are split by line, - for one embedding, word and numbers split by space - Example:: - - word_1 float_1 float_2 ... float_emb_dim - word_2 float_1 float_2 ... float_emb_dim - ... + def load_with_vocab(embed_filepath, vocab, dtype=np.float32, normalize=True, error='ignore'): """ - emb = {} - with open(emb_file, 'r', encoding='utf-8') as f: - for line in f: - line = list(filter(lambda w: len(w) > 0, line.strip().split(' '))) - if len(line) > 2: - emb[line[0]] = torch.Tensor(list(map(float, line[1:]))) - return emb - - @staticmethod - def _load_pretrain(emb_file, emb_type): - """Read txt data from embedding file and convert to np.array as pre-trained embedding - - :param str emb_file: the pre-trained embedding file path - :param str emb_type: the pre-trained embedding data format - :return: a dict of ``{str: np.array}`` + 从embed_filepath这个预训练的词向量中抽取出vocab这个词表的词的embedding。EmbedLoader将自动判断embed_filepath是 + word2vec(第一行只有两个元素)还是glove格式的数据。 + + :param str embed_filepath: 预训练的embedding的路径。 + :param vocab: 词表 :class:`~fastNLP.Vocabulary` 类型，读取出现在vocab中的词的embedding。 + 没有出现在vocab中的词的embedding将通过找到的词的embedding的正态分布采样出来，以使得整个Embedding是同分布的。 + :param dtype: 读出的embedding的类型 + :param bool normalize: 是否将每个vector归一化到norm为1 + :param str error: `ignore` , `strict` ; 如果 `ignore` ，错误将自动跳过; 如果 `strict` , 错误将抛出。 + 这里主要可能出错的地方在于词表有空行或者词表出现了维度不一致。 + :return numpy.ndarray: shape为 [len(vocab), dimension], dimension由pretrain的embedding决定。 """ - if emb_type == 'glove': - return EmbedLoader._load_glove(emb_file) - else: - raise Exception("embedding type {} not support yet".format(emb_type)) - + assert isinstance(vocab, Vocabulary), "Only fastNLP.Vocabulary is supported." + if not os.path.exists(embed_filepath): + raise FileNotFoundError("`{}` does not exist.".format(embed_filepath)) + with open(embed_filepath, 'r', encoding='utf-8') as f: + hit_flags = np.zeros(len(vocab), dtype=bool) + line = f.readline().strip() + parts = line.split() + start_idx = 0 + if len(parts) == 2: + dim = int(parts[1]) + start_idx += 1 + else: + dim = len(parts) - 1 + f.seek(0) + matrix = np.random.randn(len(vocab), dim).astype(dtype) + for idx, line in enumerate(f, start_idx): + try: + parts = line.strip().split() + if parts[0] in vocab: + index = vocab.to_index(parts[0]) + matrix[index] = np.fromstring(' '.join(parts[1:]), sep=' ', dtype=dtype, count=dim) + hit_flags[index] = True + except Exception as e: + if error == 'ignore': + warnings.warn("Error occurred at the {} line.".format(idx)) + else: + print("Error occurred at the {} line.".format(idx)) + raise e + total_hits = sum(hit_flags) + print("Found {} out of {} words in the pre-training embedding.".format(total_hits, len(vocab))) + found_vectors = matrix[hit_flags] + if len(found_vectors) != 0: + mean = np.mean(found_vectors, axis=0, keepdims=True) + std = np.std(found_vectors, axis=0, keepdims=True) + unfound_vec_num = len(vocab) - total_hits + r_vecs = np.random.randn(unfound_vec_num, dim).astype(dtype) * std + mean + matrix[hit_flags == False] = r_vecs + + if normalize: + matrix /= np.linalg.norm(matrix, axis=1, keepdims=True) + + return matrix + @staticmethod - def load_embedding(emb_dim, emb_file, emb_type, vocab): - """Load the pre-trained embedding and combine with the given dictionary. - - :param int emb_dim: the dimension of the embedding. Should be the same as pre-trained embedding. - :param str emb_file: the pre-trained embedding file path. - :param str emb_type: the pre-trained embedding format, support glove now - :param Vocabulary vocab: a mapping from word to index, can be provided by user or built from pre-trained embedding - :return (embedding_tensor, vocab): - embedding_tensor - Tensor of shape (len(word_dict), emb_dim); - vocab - input vocab or vocab built by pre-train - + def load_without_vocab(embed_filepath, dtype=np.float32, padding='', unknown='', normalize=True, + error='ignore'): """ - pretrain = EmbedLoader._load_pretrain(emb_file, emb_type) - if vocab is None: - # build vocabulary from pre-trained embedding - vocab = Vocabulary() - for w in pretrain.keys(): - vocab.add(w) - embedding_tensor = torch.randn(len(vocab), emb_dim) - for w, v in pretrain.items(): - if len(v.shape) > 1 or emb_dim != v.shape[0]: - raise ValueError( - "Pretrained embedding dim is {}. Dimension dismatched. Required {}".format(v.shape, (emb_dim,))) - if vocab.has_word(w): - embedding_tensor[vocab[w]] = v - return embedding_tensor, vocab - - @staticmethod - def parse_glove_line(line): - line = line.split() - if len(line) <= 2: - raise RuntimeError("something goes wrong in parsing glove embedding") - return line[0], line[1:] - - @staticmethod - def str_list_2_vec(line): - try: - return torch.Tensor(list(map(float, line))) - except Exception: - raise RuntimeError("something goes wrong in parsing glove embedding") - - - @staticmethod - def fast_load_embedding(emb_dim, emb_file, vocab): - """Fast load the pre-trained embedding and combine with the given dictionary. - This loading method uses line-by-line operation. - - :param int emb_dim: the dimension of the embedding. Should be the same as pre-trained embedding. - :param str emb_file: the pre-trained embedding file path. - :param Vocabulary vocab: a mapping from word to index, can be provided by user or built from pre-trained embedding - :return embedding_matrix: numpy.ndarray - + 从embed_filepath中读取预训练的word vector。根据预训练的词表读取embedding并生成一个对应的Vocabulary。 + + :param str embed_filepath: 预训练的embedding的路径。 + :param dtype: 读出的embedding的类型 + :param str padding: the padding tag for vocabulary. + :param str unknown: the unknown tag for vocabulary. + :param bool normalize: 是否将每个vector归一化到norm为1 + :param str error: `ignore` , `strict` ; 如果 `ignore` ，错误将自动跳过; 如果 `strict` , 错误将抛出。这里主要可能出错的地 + 方在于词表有空行或者词表出现了维度不一致。 + :return numpy.ndarray: shape为 [len(vocab), dimension], dimension由pretrain的embedding决定。 + :return numpy.ndarray: Vocabulary Embedding的shape是[词表大小+x, 词表维度], "词表大小+x"是由于最终的大小还取决与 + 是否使用padding, 以及unknown有没有在词表中找到对应的词。 Vocabulary中的词的顺序与Embedding的顺序是一一对应的。 """ - if vocab is None: - raise RuntimeError("You must provide a vocabulary.") - embedding_matrix = np.zeros(shape=(len(vocab), emb_dim)) - hit_flags = np.zeros(shape=(len(vocab),), dtype=int) - with open(emb_file, "r", encoding="utf-8") as f: - for line in f: - word, vector = EmbedLoader.parse_glove_line(line) - if word in vocab: - vector = EmbedLoader.str_list_2_vec(vector) - if len(vector.shape) > 1 or emb_dim != vector.shape[0]: - raise ValueError("Pre-trained embedding dim is {}. Expect {}.".format(vector.shape, (emb_dim,))) - embedding_matrix[vocab[word]] = vector - hit_flags[vocab[word]] = 1 - - if np.sum(hit_flags) < len(vocab): - # some words from vocab are missing in pre-trained embedding - # we normally sample each dimension - vocab_embed = embedding_matrix[np.where(hit_flags)] - sampled_vectors = np.random.normal(vocab_embed.mean(axis=0), vocab_embed.std(axis=0), - size=(len(vocab) - np.sum(hit_flags), emb_dim)) - embedding_matrix[np.where(1 - hit_flags)] = sampled_vectors - return embedding_matrix + vocab = Vocabulary(padding=padding, unknown=unknown) + vec_dict = {} + found_unknown = False + found_pad = False + + with open(embed_filepath, 'r', encoding='utf-8') as f: + line = f.readline() + start = 1 + dim = -1 + if len(line.strip().split()) != 2: + f.seek(0) + start = 0 + for idx, line in enumerate(f, start=start): + try: + parts = line.strip().split() + word = parts[0] + if dim == -1: + dim = len(parts) - 1 + vec = np.fromstring(' '.join(parts[1:]), sep=' ', dtype=dtype, count=dim) + vec_dict[word] = vec + vocab.add_word(word) + if unknown is not None and unknown == word: + found_unknown = True + if found_pad is not None and padding == word: + found_pad = True + except Exception as e: + if error == 'ignore': + warnings.warn("Error occurred at the {} line.".format(idx)) + pass + else: + print("Error occurred at the {} line.".format(idx)) + raise e + if dim == -1: + raise RuntimeError("{} is an empty file.".format(embed_filepath)) + matrix = np.random.randn(len(vocab), dim).astype(dtype) + if (unknown is not None and not found_unknown) or (padding is not None and not found_pad): + start_idx = 0 + if padding is not None: + start_idx += 1 + if unknown is not None: + start_idx += 1 + + mean = np.mean(matrix[start_idx:], axis=0, keepdims=True) + std = np.std(matrix[start_idx:], axis=0, keepdims=True) + if (unknown is not None and not found_unknown): + matrix[start_idx - 1] = np.random.randn(1, dim).astype(dtype) * std + mean + if (padding is not None and not found_pad): + matrix[0] = np.random.randn(1, dim).astype(dtype) * std + mean + + for key, vec in vec_dict.items(): + index = vocab.to_index(key) + matrix[index] = vec + + if normalize: + matrix /= np.linalg.norm(matrix, axis=1, keepdims=True) + + return matrix, vocab diff --git a/fastNLP/io/file_reader.py b/fastNLP/io/file_reader.py new file mode 100644 index 00000000..5963bb56 --- /dev/null +++ b/fastNLP/io/file_reader.py @@ -0,0 +1,118 @@ +""" +此模块用于给其它模块提供读取文件的函数，没有为用户提供 API +""" +import json + + +def _read_csv(path, encoding='utf-8', headers=None, sep=',', dropna=True): + """ + Construct a generator to read csv items. + + :param path: file path + :param encoding: file's encoding, default: utf-8 + :param headers: file's headers, if None, make file's first line as headers. default: None + :param sep: separator for each column. default: ',' + :param dropna: weather to ignore and drop invalid data, + :if False, raise ValueError when reading invalid data. default: True + :return: generator, every time yield (line number, csv item) + """ + with open(path, 'r', encoding=encoding) as f: + start_idx = 0 + if headers is None: + headers = f.readline().rstrip('\r\n') + headers = headers.split(sep) + start_idx += 1 + elif not isinstance(headers, (list, tuple)): + raise TypeError("headers should be list or tuple, not {}." \ + .format(type(headers))) + for line_idx, line in enumerate(f, start_idx): + contents = line.rstrip('\r\n').split(sep) + if len(contents) != len(headers): + if dropna: + continue + else: + raise ValueError("Line {} has {} parts, while header has {} parts." \ + .format(line_idx, len(contents), len(headers))) + _dict = {} + for header, content in zip(headers, contents): + _dict[header] = content + yield line_idx, _dict + + +def _read_json(path, encoding='utf-8', fields=None, dropna=True): + """ + Construct a generator to read json items. + + :param path: file path + :param encoding: file's encoding, default: utf-8 + :param fields: json object's fields that needed, if None, all fields are needed. default: None + :param dropna: weather to ignore and drop invalid data, + :if False, raise ValueError when reading invalid data. default: True + :return: generator, every time yield (line number, json item) + """ + if fields: + fields = set(fields) + with open(path, 'r', encoding=encoding) as f: + for line_idx, line in enumerate(f): + data = json.loads(line) + if fields is None: + yield line_idx, data + continue + _res = {} + for k, v in data.items(): + if k in fields: + _res[k] = v + if len(_res) < len(fields): + if dropna: + continue + else: + raise ValueError('invalid instance at line: {}'.format(line_idx)) + yield line_idx, _res + + +def _read_conll(path, encoding='utf-8', indexes=None, dropna=True): + """ + Construct a generator to read conll items. + + :param path: file path + :param encoding: file's encoding, default: utf-8 + :param indexes: conll object's column indexes that needed, if None, all columns are needed. default: None + :param dropna: weather to ignore and drop invalid data, + :if False, raise ValueError when reading invalid data. default: True + :return: generator, every time yield (line number, conll item) + """ + def parse_conll(sample): + sample = list(map(list, zip(*sample))) + sample = [sample[i] for i in indexes] + for f in sample: + if len(f) <= 0: + raise ValueError('empty field') + return sample + with open(path, 'r', encoding=encoding) as f: + sample = [] + start = next(f) + if '-DOCSTART-' not in start: + sample.append(start.split()) + for line_idx, line in enumerate(f, 1): + if line.startswith('\n'): + if len(sample): + try: + res = parse_conll(sample) + sample = [] + yield line_idx, res + except Exception as e: + if dropna: + continue + raise ValueError('invalid instance at line: {}'.format(line_idx)) + elif line.startswith('#'): + continue + else: + sample.append(line.split()) + if len(sample) > 0: + try: + res = parse_conll(sample) + yield line_idx, res + except Exception as e: + if dropna: + return + raise ValueError('invalid instance at line: {}'.format(line_idx)) diff --git a/fastNLP/io/logger.py b/fastNLP/io/logger.py deleted file mode 100644 index 9e9730db..00000000 --- a/fastNLP/io/logger.py +++ /dev/null @@ -1,35 +0,0 @@ -import logging -import os - - -def create_logger(logger_name, log_path, log_format=None, log_level=logging.INFO): - """Create a logger. - - :param str logger_name: - :param str log_path: - :param log_format: - :param log_level: - :return: logger - - To use a logger:: - - logger.debug("this is a debug message") - logger.info("this is a info message") - logger.warning("this is a warning message") - logger.error("this is an error message") - """ - logger = logging.getLogger(logger_name) - logger.setLevel(log_level) - if log_path is None: - handler = logging.StreamHandler() - else: - os.stat(os.path.dirname(os.path.abspath(log_path))) - handler = logging.FileHandler(log_path) - handler.setLevel(log_level) - if log_format is None: - log_format = "[%(asctime)s %(name)-13s %(levelname)s %(process)d %(thread)d " \ - "%(filename)s:%(lineno)-5d] %(message)s" - formatter = logging.Formatter(log_format) - handler.setFormatter(formatter) - logger.addHandler(handler) - return logger diff --git a/fastNLP/io/model_io.py b/fastNLP/io/model_io.py index 422eb919..ffaa4ef5 100644 --- a/fastNLP/io/model_io.py +++ b/fastNLP/io/model_io.py @@ -1,53 +1,72 @@ +""" +用于载入和保存模型 +""" +__all__ = [ + "ModelLoader", + "ModelSaver" +] + import torch -from fastNLP.io.base_loader import BaseLoader +from .base_loader import BaseLoader class ModelLoader(BaseLoader): """ - Loader for models. - """ + 别名：:class:`fastNLP.io.ModelLoader` :class:`fastNLP.io.model_io.ModelLoader` + 用于读取模型 + """ + def __init__(self): super(ModelLoader, self).__init__() - + @staticmethod def load_pytorch(empty_model, model_path): - """Load model parameters from ".pkl" files into the empty PyTorch model. + """ + 从 ".pkl" 文件读取 PyTorch 模型 - :param empty_model: a PyTorch model with initialized parameters. - :param str model_path: the path to the saved model. + :param empty_model: 初始化参数的 PyTorch 模型 + :param str model_path: 模型保存的路径 """ empty_model.load_state_dict(torch.load(model_path)) - + @staticmethod def load_pytorch_model(model_path): - """Load the entire model. + """ + 读取整个模型 - :param str model_path: the path to the saved model. + :param str model_path: 模型保存的路径 """ return torch.load(model_path) class ModelSaver(object): - """Save a model + """ + 别名：:class:`fastNLP.io.ModelSaver` :class:`fastNLP.io.model_io.ModelSaver` - :param str save_path: the path to the saving directory. - Example:: + 用于保存模型 + + Example:: - saver = ModelSaver("./save/model_ckpt_100.pkl") - saver.save_pytorch(model) + saver = ModelSaver("./save/model_ckpt_100.pkl") + saver.save_pytorch(model) """ - + def __init__(self, save_path): - self.save_path = save_path + """ + :param save_path: 模型保存的路径 + """ + self.save_path = save_path + def save_pytorch(self, model, param_only=True): - """Save a pytorch model into ".pkl" file. + """ + 把 PyTorch 模型存入 ".pkl" 文件 - :param model: a PyTorch model - :param bool param_only: whether only to save the model parameters or the entire model. + :param model: PyTorch 模型 + :param bool param_only: 是否只保存模型的参数（否则保存整个模型） """ if param_only is True: diff --git a/fastNLP/models/__init__.py b/fastNLP/models/__init__.py index a83c3936..14314049 100644 --- a/fastNLP/models/__init__.py +++ b/fastNLP/models/__init__.py @@ -1,6 +1,34 @@ +""" +fastNLP 在 :mod:`~fastNLP.models` 模块中内置了如 :class:`~fastNLP.models.CNNText` 、 +:class:`~fastNLP.models.SeqLabeling` 等完整的模型，以供用户直接使用。 + +.. todo:: + 这些模型的介绍（与主页一致） + + +""" +__all__ = [ + "CNNText", + + "SeqLabeling", + "AdvSeqLabel", + + "ESIM", + + "StarTransEnc", + "STSeqLabel", + "STNLICls", + "STSeqCls", + + "BiaffineParser", + "GraphParser" +] + from .base_model import BaseModel +from .bert import BertForMultipleChoice, BertForQuestionAnswering, BertForSequenceClassification, \ + BertForTokenClassification from .biaffine_parser import BiaffineParser, GraphParser -from .char_language_model import CharLM from .cnn_text_classification import CNNText -from .sequence_modeling import SeqLabeling, AdvSeqLabel +from .sequence_labeling import SeqLabeling, AdvSeqLabel from .snli import ESIM +from .star_transformer import StarTransEnc, STSeqCls, STNLICls, STSeqLabel diff --git a/fastNLP/models/base_model.py b/fastNLP/models/base_model.py index ec532014..2646d580 100644 --- a/fastNLP/models/base_model.py +++ b/fastNLP/models/base_model.py @@ -1,18 +1,18 @@ import torch -from fastNLP.modules.decoder.MLP import MLP +from ..modules.decoder.mlp import MLP class BaseModel(torch.nn.Module): """Base PyTorch model for all models. """ - + def __init__(self): super(BaseModel, self).__init__() - + def fit(self, train_data, dev_data=None, **train_args): pass - + def predict(self, *args, **kwargs): raise NotImplementedError @@ -21,9 +21,9 @@ class NaiveClassifier(BaseModel): def __init__(self, in_feature_dim, out_feature_dim): super(NaiveClassifier, self).__init__() self.mlp = MLP([in_feature_dim, in_feature_dim, out_feature_dim]) - + def forward(self, x): return {"predict": torch.sigmoid(self.mlp(x))} - + def predict(self, x): return {"predict": torch.sigmoid(self.mlp(x)) > 0.5} diff --git a/fastNLP/models/bert.py b/fastNLP/models/bert.py new file mode 100644 index 00000000..960132ad --- /dev/null +++ b/fastNLP/models/bert.py @@ -0,0 +1,293 @@ +""" +bert.py is modified from huggingface/pytorch-pretrained-BERT, which is licensed under the Apache License 2.0. + +""" +import torch +from torch import nn + +from .base_model import BaseModel +from ..core.const import Const +from ..modules.encoder import BertModel + + +class BertForSequenceClassification(BaseModel): + """BERT model for classification. + This module is composed of the BERT model with a linear layer on top of + the pooled output. + Params: + `config`: a BertConfig class instance with the configuration to build a new model. + `num_labels`: the number of classes for the classifier. Default = 2. + Inputs: + `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] + with the word token indices in the vocabulary. Items in the batch should begin with the special "CLS" token. (see the tokens preprocessing logic in the scripts + `extract_features.py`, `run_classifier.py` and `run_squad.py`) + `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token + types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to + a `sentence B` token (see BERT paper for more details). + `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices + selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max + input sequence length in the current batch. It's the mask that we typically use for attention when + a batch has varying length sentences. + `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] + with indices selected in [0, ..., num_labels]. + Outputs: + if `labels` is not `None`: + Outputs the CrossEntropy classification loss of the output with the labels. + if `labels` is `None`: + Outputs the classification logits of shape [batch_size, num_labels]. + Example usage: + ```python + # Already been converted into WordPiece token ids + input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) + input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) + token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) + config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, + num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) + num_labels = 2 + model = BertForSequenceClassification(config, num_labels) + logits = model(input_ids, token_type_ids, input_mask) + ``` + """ + def __init__(self, config, num_labels, bert_dir): + super(BertForSequenceClassification, self).__init__() + self.num_labels = num_labels + self.bert = BertModel.from_pretrained(bert_dir) + self.dropout = nn.Dropout(config.hidden_dropout_prob) + self.classifier = nn.Linear(config.hidden_size, num_labels) + + def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None): + _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False) + pooled_output = self.dropout(pooled_output) + logits = self.classifier(pooled_output) + + if labels is not None: + loss_fct = nn.CrossEntropyLoss() + loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) + return {Const.OUTPUT: logits, Const.LOSS: loss} + else: + return {Const.OUTPUT: logits} + + def predict(self, input_ids, token_type_ids=None, attention_mask=None): + logits = self.forward(input_ids, token_type_ids, attention_mask) + return {Const.OUTPUT: torch.argmax(logits, dim=-1)} + + +class BertForMultipleChoice(BaseModel): + """BERT model for multiple choice tasks. + This module is composed of the BERT model with a linear layer on top of + the pooled output. + Params: + `config`: a BertConfig class instance with the configuration to build a new model. + `num_choices`: the number of classes for the classifier. Default = 2. + Inputs: + `input_ids`: a torch.LongTensor of shape [batch_size, num_choices, sequence_length] + with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts + `extract_features.py`, `run_classifier.py` and `run_squad.py`) + `token_type_ids`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] + with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` + and type 1 corresponds to a `sentence B` token (see BERT paper for more details). + `attention_mask`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] with indices + selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max + input sequence length in the current batch. It's the mask that we typically use for attention when + a batch has varying length sentences. + `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] + with indices selected in [0, ..., num_choices]. + Outputs: + if `labels` is not `None`: + Outputs the CrossEntropy classification loss of the output with the labels. + if `labels` is `None`: + Outputs the classification logits of shape [batch_size, num_labels]. + Example usage: + ```python + # Already been converted into WordPiece token ids + input_ids = torch.LongTensor([[[31, 51, 99], [15, 5, 0]], [[12, 16, 42], [14, 28, 57]]]) + input_mask = torch.LongTensor([[[1, 1, 1], [1, 1, 0]],[[1,1,0], [1, 0, 0]]]) + token_type_ids = torch.LongTensor([[[0, 0, 1], [0, 1, 0]],[[0, 1, 1], [0, 0, 1]]]) + config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, + num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) + num_choices = 2 + model = BertForMultipleChoice(config, num_choices, bert_dir) + logits = model(input_ids, token_type_ids, input_mask) + ``` + """ + def __init__(self, config, num_choices, bert_dir): + super(BertForMultipleChoice, self).__init__() + self.num_choices = num_choices + self.bert = BertModel.from_pretrained(bert_dir) + self.dropout = nn.Dropout(config.hidden_dropout_prob) + self.classifier = nn.Linear(config.hidden_size, 1) + + def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None): + flat_input_ids = input_ids.view(-1, input_ids.size(-1)) + flat_token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) + flat_attention_mask = attention_mask.view(-1, attention_mask.size(-1)) + _, pooled_output = self.bert(flat_input_ids, flat_token_type_ids, flat_attention_mask, output_all_encoded_layers=False) + pooled_output = self.dropout(pooled_output) + logits = self.classifier(pooled_output) + reshaped_logits = logits.view(-1, self.num_choices) + + if labels is not None: + loss_fct = nn.CrossEntropyLoss() + loss = loss_fct(reshaped_logits, labels) + return {Const.OUTPUT: reshaped_logits, Const.LOSS: loss} + else: + return {Const.OUTPUT: reshaped_logits} + + def predict(self, input_ids, token_type_ids=None, attention_mask=None): + logits = self.forward(input_ids, token_type_ids, attention_mask)[Const.OUTPUT] + return {Const.OUTPUT: torch.argmax(logits, dim=-1)} + + +class BertForTokenClassification(BaseModel): + """BERT model for token-level classification. + This module is composed of the BERT model with a linear layer on top of + the full hidden state of the last layer. + Params: + `config`: a BertConfig class instance with the configuration to build a new model. + `num_labels`: the number of classes for the classifier. Default = 2. + `bert_dir`: a dir which contains the bert parameters within file `pytorch_model.bin` + Inputs: + `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] + with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts + `extract_features.py`, `run_classifier.py` and `run_squad.py`) + `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token + types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to + a `sentence B` token (see BERT paper for more details). + `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices + selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max + input sequence length in the current batch. It's the mask that we typically use for attention when + a batch has varying length sentences. + `labels`: labels for the classification output: torch.LongTensor of shape [batch_size, sequence_length] + with indices selected in [0, ..., num_labels]. + Outputs: + if `labels` is not `None`: + Outputs the CrossEntropy classification loss of the output with the labels. + if `labels` is `None`: + Outputs the classification logits of shape [batch_size, sequence_length, num_labels]. + Example usage: + ```python + # Already been converted into WordPiece token ids + input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) + input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) + token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) + config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, + num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) + num_labels = 2 + bert_dir = 'your-bert-file-dir' + model = BertForTokenClassification(config, num_labels, bert_dir) + logits = model(input_ids, token_type_ids, input_mask) + ``` + """ + def __init__(self, config, num_labels, bert_dir): + super(BertForTokenClassification, self).__init__() + self.num_labels = num_labels + self.bert = BertModel.from_pretrained(bert_dir) + self.dropout = nn.Dropout(config.hidden_dropout_prob) + self.classifier = nn.Linear(config.hidden_size, num_labels) + + def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None): + sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False) + sequence_output = self.dropout(sequence_output) + logits = self.classifier(sequence_output) + + if labels is not None: + loss_fct = nn.CrossEntropyLoss() + # Only keep active parts of the loss + if attention_mask is not None: + active_loss = attention_mask.view(-1) == 1 + active_logits = logits.view(-1, self.num_labels)[active_loss] + active_labels = labels.view(-1)[active_loss] + loss = loss_fct(active_logits, active_labels) + else: + loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) + return {Const.OUTPUT: logits, Const.LOSS: loss} + else: + return {Const.OUTPUT: logits} + + def predict(self, input_ids, token_type_ids=None, attention_mask=None): + logits = self.forward(input_ids, token_type_ids, attention_mask)[Const.OUTPUT] + return {Const.OUTPUT: torch.argmax(logits, dim=-1)} + + +class BertForQuestionAnswering(BaseModel): + """BERT model for Question Answering (span extraction). + This module is composed of the BERT model with a linear layer on top of + the sequence output that computes start_logits and end_logits + Params: + `config`: a BertConfig class instance with the configuration to build a new model. + `bert_dir`: a dir which contains the bert parameters within file `pytorch_model.bin` + Inputs: + `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] + with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts + `extract_features.py`, `run_classifier.py` and `run_squad.py`) + `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token + types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to + a `sentence B` token (see BERT paper for more details). + `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices + selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max + input sequence length in the current batch. It's the mask that we typically use for attention when + a batch has varying length sentences. + `start_positions`: position of the first token for the labeled span: torch.LongTensor of shape [batch_size]. + Positions are clamped to the length of the sequence and position outside of the sequence are not taken + into account for computing the loss. + `end_positions`: position of the last token for the labeled span: torch.LongTensor of shape [batch_size]. + Positions are clamped to the length of the sequence and position outside of the sequence are not taken + into account for computing the loss. + Outputs: + if `start_positions` and `end_positions` are not `None`: + Outputs the total_loss which is the sum of the CrossEntropy loss for the start and end token positions. + if `start_positions` or `end_positions` is `None`: + Outputs a tuple of start_logits, end_logits which are the logits respectively for the start and end + position tokens of shape [batch_size, sequence_length]. + Example usage: + ```python + # Already been converted into WordPiece token ids + input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) + input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) + token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) + config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, + num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) + bert_dir = 'your-bert-file-dir' + model = BertForQuestionAnswering(config, bert_dir) + start_logits, end_logits = model(input_ids, token_type_ids, input_mask) + ``` + """ + def __init__(self, config, bert_dir): + super(BertForQuestionAnswering, self).__init__() + self.bert = BertModel.from_pretrained(bert_dir) + # TODO check with Google if it's normal there is no dropout on the token classifier of SQuAD in the TF version + # self.dropout = nn.Dropout(config.hidden_dropout_prob) + self.qa_outputs = nn.Linear(config.hidden_size, 2) + + def forward(self, input_ids, token_type_ids=None, attention_mask=None, start_positions=None, end_positions=None): + sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False) + logits = self.qa_outputs(sequence_output) + start_logits, end_logits = logits.split(1, dim=-1) + start_logits = start_logits.squeeze(-1) + end_logits = end_logits.squeeze(-1) + + if start_positions is not None and end_positions is not None: + # If we are on multi-GPU, split add a dimension + if len(start_positions.size()) > 1: + start_positions = start_positions.squeeze(-1) + if len(end_positions.size()) > 1: + end_positions = end_positions.squeeze(-1) + # sometimes the start/end positions are outside our model inputs, we ignore these terms + ignored_index = start_logits.size(1) + start_positions.clamp_(0, ignored_index) + end_positions.clamp_(0, ignored_index) + + loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index) + start_loss = loss_fct(start_logits, start_positions) + end_loss = loss_fct(end_logits, end_positions) + total_loss = (start_loss + end_loss) / 2 + return {Const.OUTPUTS(0): start_logits, Const.OUTPUTS(1): end_logits, Const.LOSS: total_loss} + else: + return {Const.OUTPUTS(0): start_logits, Const.OUTPUTS(1): end_logits} + + def predict(self, input_ids, token_type_ids=None, attention_mask=None, **kwargs): + logits = self.forward(input_ids, token_type_ids, attention_mask) + start_logits = logits[Const.OUTPUTS(0)] + end_logits = logits[Const.OUTPUTS(1)] + return {Const.OUTPUTS(0): torch.argmax(start_logits, dim=-1), + Const.OUTPUTS(1): torch.argmax(end_logits, dim=-1)} diff --git a/fastNLP/models/biaffine_parser.py b/fastNLP/models/biaffine_parser.py index fb687301..8533e7af 100644 --- a/fastNLP/models/biaffine_parser.py +++ b/fastNLP/models/biaffine_parser.py @@ -1,19 +1,31 @@ -import copy +""" +Biaffine Dependency Parser 的 Pytorch 实现. +""" +__all__ = [ + "BiaffineParser", + "GraphParser" +] + import numpy as np import torch +import torch.nn as nn +import torch.nn.functional as F + from collections import defaultdict -from torch import nn -from torch.nn import functional as F -from fastNLP.modules.utils import initial_parameter -from fastNLP.modules.encoder.variational_rnn import VarLSTM -from fastNLP.modules.dropout import TimestepDropout -from fastNLP.models.base_model import BaseModel -from fastNLP.modules.utils import seq_mask -from fastNLP.core.losses import LossFunc -from fastNLP.core.metrics import MetricBase -from fastNLP.core.utils import seq_lens_to_masks - -def mst(scores): + +from ..core.const import Const as C +from ..core.losses import LossFunc +from ..core.metrics import MetricBase +from ..modules.dropout import TimestepDropout +from ..modules.encoder.transformer import TransformerEncoder +from ..modules.encoder.variational_rnn import VarLSTM +from ..modules.utils import initial_parameter +from ..modules.utils import get_embeddings +from .base_model import BaseModel +from ..core.utils import seq_len_to_mask + + +def _mst(scores): """ with some modification to support parser output for MST decoding https://github.com/tdozat/Parser/blob/0739216129cd39d69997d28cbc4133b360ea3934/lib/models/nn.py#L692 @@ -39,7 +51,7 @@ def mst(scores): scores[roots, new_heads] / root_scores)] heads[roots] = new_heads heads[new_root] = 0 - + edges = defaultdict(set) vertices = set((0,)) for dep, head in enumerate(heads[tokens]): @@ -68,7 +80,7 @@ def mst(scores): heads[changed_cycle] = new_head edges[new_head].add(changed_cycle) edges[old_head].remove(changed_cycle) - + return heads @@ -83,7 +95,7 @@ def _find_cycle(vertices, edges): _lowlinks = {} _onstack = defaultdict(lambda: False) _SCCs = [] - + def _strongconnect(v): nonlocal _index _indices[v] = _index @@ -91,38 +103,49 @@ def _find_cycle(vertices, edges): _index += 1 _stack.append(v) _onstack[v] = True - + for w in edges[v]: if w not in _indices: _strongconnect(w) _lowlinks[v] = min(_lowlinks[v], _lowlinks[w]) elif _onstack[w]: _lowlinks[v] = min(_lowlinks[v], _indices[w]) - + if _lowlinks[v] == _indices[v]: SCC = set() while True: w = _stack.pop() _onstack[w] = False SCC.add(w) - if not(w != v): + if not (w != v): break _SCCs.append(SCC) - + for v in vertices: if v not in _indices: _strongconnect(v) - + return [SCC for SCC in _SCCs if len(SCC) > 1] class GraphParser(BaseModel): - """Graph based Parser helper class, support greedy decoding and MST(Maximum Spanning Tree) decoding """ + 基于图的parser base class, 支持贪婪解码和最大生成树解码 + """ + def __init__(self): super(GraphParser, self).__init__() + + @staticmethod + def greedy_decoder(arc_matrix, mask=None): + """ + 贪心解码方式, 输入图, 输出贪心解码的parsing结果, 不保证合法的构成树 - def _greedy_decoder(self, arc_matrix, mask=None): + :param arc_matrix: [batch, seq_len, seq_len] 输入图矩阵 + :param mask: [batch, seq_len] 输入图的padding mask, 有内容的部分为 1, 否则为 0. + 若为 ``None`` 时, 默认为全1向量. Default: ``None`` + :return heads: [batch, seq_len] 每个元素在树中对应的head(parent)预测结果 + """ _, seq_len, _ = arc_matrix.shape matrix = arc_matrix + torch.diag(arc_matrix.new(seq_len).fill_(-np.inf)) flip_mask = (mask == 0).byte() @@ -131,24 +154,37 @@ class GraphParser(BaseModel): if mask is not None: heads *= mask.long() return heads + + @staticmethod + def mst_decoder(arc_matrix, mask=None): + """ + 用最大生成树算法, 计算parsing结果, 保证输出合法的树结构 - def _mst_decoder(self, arc_matrix, mask=None): + :param arc_matrix: [batch, seq_len, seq_len] 输入图矩阵 + :param mask: [batch, seq_len] 输入图的padding mask, 有内容的部分为 1, 否则为 0. + 若为 ``None`` 时, 默认为全1向量. Default: ``None`` + :return heads: [batch, seq_len] 每个元素在树中对应的head(parent)预测结果 + """ batch_size, seq_len, _ = arc_matrix.shape matrix = arc_matrix.clone() ans = matrix.new_zeros(batch_size, seq_len).long() lens = (mask.long()).sum(1) if mask is not None else torch.zeros(batch_size) + seq_len - batch_idx = torch.arange(batch_size, dtype=torch.long, device=lens.device) for i, graph in enumerate(matrix): len_i = lens[i] - ans[i, :len_i] = torch.as_tensor(mst(graph.detach()[:len_i, :len_i].cpu().numpy()), device=ans.device) + ans[i, :len_i] = torch.as_tensor(_mst(graph.detach()[:len_i, :len_i].cpu().numpy()), device=ans.device) if mask is not None: ans *= mask.long() return ans class ArcBiaffine(nn.Module): - """helper module for Biaffine Dependency Parser predicting arc """ + Biaffine Dependency Parser 的子模块, 用于构建预测边的图 + + :param hidden_size: 输入的特征维度 + :param bias: 是否使用bias. Default: ``True`` + """ + def __init__(self, hidden_size, bias=True): super(ArcBiaffine, self).__init__() self.U = nn.Parameter(torch.Tensor(hidden_size, hidden_size), requires_grad=True) @@ -158,13 +194,13 @@ class ArcBiaffine(nn.Module): else: self.register_parameter("bias", None) initial_parameter(self) - + def forward(self, head, dep): """ - :param head arc-head tensor = [batch, length, emb_dim] - :param dep arc-dependent tensor = [batch, length, emb_dim] - :return output tensor = [bacth, length, length] + :param head: arc-head tensor [batch, length, hidden] + :param dep: arc-dependent tensor [batch, length, hidden] + :return output: tensor [bacth, length, length] """ output = dep.matmul(self.U) output = output.bmm(head.transpose(-1, -2)) @@ -174,82 +210,124 @@ class ArcBiaffine(nn.Module): class LabelBilinear(nn.Module): - """helper module for Biaffine Dependency Parser predicting label """ + Biaffine Dependency Parser 的子模块, 用于构建预测边类别的图 + + :param in1_features: 输入的特征1维度 + :param in2_features: 输入的特征2维度 + :param num_label: 边类别的个数 + :param bias: 是否使用bias. Default: ``True`` + """ + def __init__(self, in1_features, in2_features, num_label, bias=True): super(LabelBilinear, self).__init__() self.bilinear = nn.Bilinear(in1_features, in2_features, num_label, bias=bias) self.lin = nn.Linear(in1_features + in2_features, num_label, bias=False) - + def forward(self, x1, x2): + """ + + :param x1: [batch, seq_len, hidden] 输入特征1, 即label-head + :param x2: [batch, seq_len, hidden] 输入特征2, 即label-dep + :return output: [batch, seq_len, num_cls] 每个元素对应类别的概率图 + """ output = self.bilinear(x1, x2) output += self.lin(torch.cat([x1, x2], dim=2)) return output + class BiaffineParser(GraphParser): - """Biaffine Dependency Parser implemantation. - refer to ` Deep Biaffine Attention for Neural Dependency Parsing (Dozat and Manning, 2016) - `_ . """ + 别名：:class:`fastNLP.models.BiaffineParser` :class:`fastNLP.models.baffine_parser.BiaffineParser` + + Biaffine Dependency Parser 实现. + 论文参考 `Deep Biaffine Attention for Neural Dependency Parsing (Dozat and Manning, 2016) `_ . + + :param init_embed: 单词词典, 可以是 tuple, 包括(num_embedings, embedding_dim), 即 + embedding的大小和每个词的维度. 也可以传入 nn.Embedding 对象, + 此时就以传入的对象作为embedding + :param pos_vocab_size: part-of-speech 词典大小 + :param pos_emb_dim: part-of-speech 向量维度 + :param num_label: 边的类别个数 + :param rnn_layers: rnn encoder的层数 + :param rnn_hidden_size: rnn encoder 的隐状态维度 + :param arc_mlp_size: 边预测的MLP维度 + :param label_mlp_size: 类别预测的MLP维度 + :param dropout: dropout概率. + :param encoder: encoder类别, 可选 ('lstm', 'var-lstm', 'transformer'). Default: lstm + :param use_greedy_infer: 是否在inference时使用贪心算法. + 若 ``False`` , 使用更加精确但相对缓慢的MST算法. Default: ``False`` + """ + def __init__(self, - word_vocab_size, - word_emb_dim, - pos_vocab_size, - pos_emb_dim, - num_label, - word_hid_dim=100, - pos_hid_dim=100, - rnn_layers=1, - rnn_hidden_size=200, - arc_mlp_size=100, - label_mlp_size=100, - dropout=0.3, - use_var_lstm=False, - use_greedy_infer=False): - + init_embed, + pos_vocab_size, + pos_emb_dim, + num_label, + rnn_layers=1, + rnn_hidden_size=200, + arc_mlp_size=100, + label_mlp_size=100, + dropout=0.3, + encoder='lstm', + use_greedy_infer=False): super(BiaffineParser, self).__init__() rnn_out_size = 2 * rnn_hidden_size - self.word_embedding = nn.Embedding(num_embeddings=word_vocab_size, embedding_dim=word_emb_dim) + word_hid_dim = pos_hid_dim = rnn_hidden_size + self.word_embedding = get_embeddings(init_embed) + word_emb_dim = self.word_embedding.embedding_dim self.pos_embedding = nn.Embedding(num_embeddings=pos_vocab_size, embedding_dim=pos_emb_dim) self.word_fc = nn.Linear(word_emb_dim, word_hid_dim) self.pos_fc = nn.Linear(pos_emb_dim, pos_hid_dim) self.word_norm = nn.LayerNorm(word_hid_dim) self.pos_norm = nn.LayerNorm(pos_hid_dim) - self.use_var_lstm = use_var_lstm - if use_var_lstm: - self.lstm = VarLSTM(input_size=word_hid_dim + pos_hid_dim, - hidden_size=rnn_hidden_size, - num_layers=rnn_layers, - bias=True, - batch_first=True, - input_dropout=dropout, - hidden_dropout=dropout, - bidirectional=True) + self.encoder_name = encoder + self.max_len = 512 + if encoder == 'var-lstm': + self.encoder = VarLSTM(input_size=word_hid_dim + pos_hid_dim, + hidden_size=rnn_hidden_size, + num_layers=rnn_layers, + bias=True, + batch_first=True, + input_dropout=dropout, + hidden_dropout=dropout, + bidirectional=True) + elif encoder == 'lstm': + self.encoder = nn.LSTM(input_size=word_hid_dim + pos_hid_dim, + hidden_size=rnn_hidden_size, + num_layers=rnn_layers, + bias=True, + batch_first=True, + dropout=dropout, + bidirectional=True) + elif encoder == 'transformer': + n_head = 16 + d_k = d_v = int(rnn_out_size / n_head) + if (d_k * n_head) != rnn_out_size: + raise ValueError('unsupported rnn_out_size: {} for transformer'.format(rnn_out_size)) + self.position_emb = nn.Embedding(num_embeddings=self.max_len, + embedding_dim=rnn_out_size, ) + self.encoder = TransformerEncoder(num_layers=rnn_layers, + model_size=rnn_out_size, + inner_size=1024, + key_size=d_k, + value_size=d_v, + num_head=n_head, + dropout=dropout, ) else: - self.lstm = nn.LSTM(input_size=word_hid_dim + pos_hid_dim, - hidden_size=rnn_hidden_size, - num_layers=rnn_layers, - bias=True, - batch_first=True, - dropout=dropout, - bidirectional=True) - - self.arc_head_mlp = nn.Sequential(nn.Linear(rnn_out_size, arc_mlp_size), - nn.LayerNorm(arc_mlp_size), - nn.ELU(), - TimestepDropout(p=dropout),) - self.arc_dep_mlp = copy.deepcopy(self.arc_head_mlp) - self.label_head_mlp = nn.Sequential(nn.Linear(rnn_out_size, label_mlp_size), - nn.LayerNorm(label_mlp_size), - nn.ELU(), - TimestepDropout(p=dropout),) - self.label_dep_mlp = copy.deepcopy(self.label_head_mlp) + raise ValueError('unsupported encoder type: {}'.format(encoder)) + + self.mlp = nn.Sequential(nn.Linear(rnn_out_size, arc_mlp_size * 2 + label_mlp_size * 2), + nn.ELU(), + TimestepDropout(p=dropout), ) + self.arc_mlp_size = arc_mlp_size + self.label_mlp_size = label_mlp_size self.arc_predictor = ArcBiaffine(arc_mlp_size, bias=True) self.label_predictor = LabelBilinear(label_mlp_size, label_mlp_size, num_label, bias=True) self.use_greedy_infer = use_greedy_infer self.reset_parameters() self.dropout = dropout - + def reset_parameters(self): for m in self.modules(): if isinstance(m, nn.Embedding): @@ -260,165 +338,210 @@ class BiaffineParser(GraphParser): else: for p in m.parameters(): nn.init.normal_(p, 0, 0.1) + + def forward(self, words1, words2, seq_len, target1=None): + """模型forward阶段 + + :param words1: [batch_size, seq_len] 输入word序列 + :param words2: [batch_size, seq_len] 输入pos序列 + :param seq_len: [batch_size, seq_len] 输入序列长度 + :param target1: [batch_size, seq_len] 输入真实标注的heads, 仅在训练阶段有效, + 用于训练label分类器. 若为 ``None`` , 使用预测的heads输入到label分类器 + Default: ``None`` + :return dict: parsing + 结果:: + + pred1: [batch_size, seq_len, seq_len] 边预测logits + pred2: [batch_size, seq_len, num_label] label预测logits + pred3: [batch_size, seq_len] heads的预测结果, 在 ``target1=None`` 时预测 - def forward(self, word_seq, pos_seq, seq_lens, gold_heads=None): - """ - :param word_seq: [batch_size, seq_len] sequence of word's indices - :param pos_seq: [batch_size, seq_len] sequence of word's indices - :param seq_lens: [batch_size, seq_len] sequence of length masks - :param gold_heads: [batch_size, seq_len] sequence of golden heads - :return dict: parsing results - arc_pred: [batch_size, seq_len, seq_len] - label_pred: [batch_size, seq_len, seq_len] - mask: [batch_size, seq_len] - head_pred: [batch_size, seq_len] if gold_heads is not provided, predicting the heads """ # prepare embeddings - batch_size, seq_len = word_seq.shape + batch_size, length = words1.shape # print('forward {} {}'.format(batch_size, seq_len)) - + # get sequence mask - mask = seq_mask(seq_lens, seq_len).long() - - word = self.word_embedding(word_seq) # [N,L] -> [N,L,C_0] - pos = self.pos_embedding(pos_seq) # [N,L] -> [N,L,C_1] - + mask = seq_len_to_mask(seq_len).long() + + word = self.word_embedding(words1) # [N,L] -> [N,L,C_0] + pos = self.pos_embedding(words2) # [N,L] -> [N,L,C_1] + word, pos = self.word_fc(word), self.pos_fc(pos) word, pos = self.word_norm(word), self.pos_norm(pos) - x = torch.cat([word, pos], dim=2) # -> [N,L,C] - del word, pos - - # lstm, extract features - sort_lens, sort_idx = torch.sort(seq_lens, dim=0, descending=True) - x = x[sort_idx] - x = nn.utils.rnn.pack_padded_sequence(x, sort_lens, batch_first=True) - feat, _ = self.lstm(x) # -> [N,L,C] - feat, _ = nn.utils.rnn.pad_packed_sequence(feat, batch_first=True) - _, unsort_idx = torch.sort(sort_idx, dim=0, descending=False) - feat = feat[unsort_idx] - + x = torch.cat([word, pos], dim=2) # -> [N,L,C] + + # encoder, extract features + if self.encoder_name.endswith('lstm'): + sort_lens, sort_idx = torch.sort(seq_len, dim=0, descending=True) + x = x[sort_idx] + x = nn.utils.rnn.pack_padded_sequence(x, sort_lens, batch_first=True) + feat, _ = self.encoder(x) # -> [N,L,C] + feat, _ = nn.utils.rnn.pad_packed_sequence(feat, batch_first=True) + _, unsort_idx = torch.sort(sort_idx, dim=0, descending=False) + feat = feat[unsort_idx] + else: + seq_range = torch.arange(length, dtype=torch.long, device=x.device)[None, :] + x = x + self.position_emb(seq_range) + feat = self.encoder(x, mask.float()) + # for arc biaffine # mlp, reduce dim - arc_dep = self.arc_dep_mlp(feat) - arc_head = self.arc_head_mlp(feat) - label_dep = self.label_dep_mlp(feat) - label_head = self.label_head_mlp(feat) - del feat - + feat = self.mlp(feat) + arc_sz, label_sz = self.arc_mlp_size, self.label_mlp_size + arc_dep, arc_head = feat[:, :, :arc_sz], feat[:, :, arc_sz:2 * arc_sz] + label_dep, label_head = feat[:, :, 2 * arc_sz:2 * arc_sz + label_sz], feat[:, :, 2 * arc_sz + label_sz:] + # biaffine arc classifier - arc_pred = self.arc_predictor(arc_head, arc_dep) # [N, L, L] - + arc_pred = self.arc_predictor(arc_head, arc_dep) # [N, L, L] + # use gold or predicted arc to predict label - if gold_heads is None or not self.training: + if target1 is None or not self.training: # use greedy decoding in training if self.training or self.use_greedy_infer: - heads = self._greedy_decoder(arc_pred, mask) + heads = self.greedy_decoder(arc_pred, mask) else: - heads = self._mst_decoder(arc_pred, mask) + heads = self.mst_decoder(arc_pred, mask) head_pred = heads else: - assert self.training # must be training mode - if gold_heads is None: - heads = self._greedy_decoder(arc_pred, mask) + assert self.training # must be training mode + if target1 is None: + heads = self.greedy_decoder(arc_pred, mask) head_pred = heads else: head_pred = None - heads = gold_heads - - batch_range = torch.arange(start=0, end=batch_size, dtype=torch.long, device=word_seq.device).unsqueeze(1) + heads = target1 + + batch_range = torch.arange(start=0, end=batch_size, dtype=torch.long, device=words1.device).unsqueeze(1) label_head = label_head[batch_range, heads].contiguous() - label_pred = self.label_predictor(label_head, label_dep) # [N, L, num_label] - res_dict = {'arc_pred': arc_pred, 'label_pred': label_pred, 'mask': mask} + label_pred = self.label_predictor(label_head, label_dep) # [N, L, num_label] + res_dict = {C.OUTPUTS(0): arc_pred, C.OUTPUTS(1): label_pred} if head_pred is not None: - res_dict['head_pred'] = head_pred + res_dict[C.OUTPUTS(2)] = head_pred return res_dict - + @staticmethod - def loss(arc_pred, label_pred, arc_true, label_true, mask): + def loss(pred1, pred2, target1, target2, seq_len): """ - Compute loss. - - :param arc_pred: [batch_size, seq_len, seq_len] - :param label_pred: [batch_size, seq_len, n_tags] - :param arc_true: [batch_size, seq_len] - :param label_true: [batch_size, seq_len] - :param mask: [batch_size, seq_len] - :return: loss value + 计算parser的loss + + :param pred1: [batch_size, seq_len, seq_len] 边预测logits + :param pred2: [batch_size, seq_len, num_label] label预测logits + :param target1: [batch_size, seq_len] 真实边的标注 + :param target2: [batch_size, seq_len] 真实类别的标注 + :param seq_len: [batch_size, seq_len] 真实目标的长度 + :return loss: scalar """ - - batch_size, seq_len, _ = arc_pred.shape + + batch_size, length, _ = pred1.shape + mask = seq_len_to_mask(seq_len) flip_mask = (mask == 0) - _arc_pred = arc_pred.clone() - _arc_pred.masked_fill_(flip_mask.unsqueeze(1), -np.inf) + _arc_pred = pred1.clone() + _arc_pred.masked_fill_(flip_mask.unsqueeze(1), -float('inf')) arc_logits = F.log_softmax(_arc_pred, dim=2) - label_logits = F.log_softmax(label_pred, dim=2) + label_logits = F.log_softmax(pred2, dim=2) batch_index = torch.arange(batch_size, device=arc_logits.device, dtype=torch.long).unsqueeze(1) - child_index = torch.arange(seq_len, device=arc_logits.device, dtype=torch.long).unsqueeze(0) - arc_loss = arc_logits[batch_index, child_index, arc_true] - label_loss = label_logits[batch_index, child_index, label_true] - - arc_loss = arc_loss[:, 1:] - label_loss = label_loss[:, 1:] - - float_mask = mask[:, 1:].float() - arc_nll = -(arc_loss*float_mask).mean() - label_nll = -(label_loss*float_mask).mean() + child_index = torch.arange(length, device=arc_logits.device, dtype=torch.long).unsqueeze(0) + arc_loss = arc_logits[batch_index, child_index, target1] + label_loss = label_logits[batch_index, child_index, target2] + + byte_mask = flip_mask.byte() + arc_loss.masked_fill_(byte_mask, 0) + label_loss.masked_fill_(byte_mask, 0) + arc_nll = -arc_loss.mean() + label_nll = -label_loss.mean() return arc_nll + label_nll + + def predict(self, words1, words2, seq_len): + """模型预测API - def predict(self, word_seq, pos_seq, seq_lens): - """ + :param words1: [batch_size, seq_len] 输入word序列 + :param words2: [batch_size, seq_len] 输入pos序列 + :param seq_len: [batch_size, seq_len] 输入序列长度 + :return dict: parsing + 结果:: + + pred1: [batch_size, seq_len] heads的预测结果 + pred2: [batch_size, seq_len, num_label] label预测logits - :param word_seq: - :param pos_seq: - :param seq_lens: - :return: arc_pred: [B, L] - label_pred: [B, L] """ - res = self(word_seq, pos_seq, seq_lens) + res = self(words1, words2, seq_len) output = {} - output['arc_pred'] = res.pop('head_pred') - _, label_pred = res.pop('label_pred').max(2) - output['label_pred'] = label_pred + output[C.OUTPUTS(0)] = res.pop(C.OUTPUTS(2)) + _, label_pred = res.pop(C.OUTPUTS(1)).max(2) + output[C.OUTPUTS(1)] = label_pred return output class ParserLoss(LossFunc): - def __init__(self, arc_pred=None, label_pred=None, arc_true=None, label_true=None): + """ + 别名：:class:`fastNLP.models.ParserLoss` :class:`fastNLP.models.baffine_parser.ParserLoss` + + 计算parser的loss + + :param pred1: [batch_size, seq_len, seq_len] 边预测logits + :param pred2: [batch_size, seq_len, num_label] label预测logits + :param target1: [batch_size, seq_len] 真实边的标注 + :param target2: [batch_size, seq_len] 真实类别的标注 + :param seq_len: [batch_size, seq_len] 真实目标的长度 + :return loss: scalar + """ + + def __init__(self, pred1=None, pred2=None, + target1=None, target2=None, + seq_len=None): super(ParserLoss, self).__init__(BiaffineParser.loss, - arc_pred=arc_pred, - label_pred=label_pred, - arc_true=arc_true, - label_true=label_true) + pred1=pred1, + pred2=pred2, + target1=target1, + target2=target2, + seq_len=seq_len) class ParserMetric(MetricBase): - def __init__(self, arc_pred=None, label_pred=None, - arc_true=None, label_true=None, seq_lens=None): + """ + 别名：:class:`fastNLP.models.ParserMetric` :class:`fastNLP.models.baffine_parser.ParserMetric` + + 评估parser的性能 + + :param pred1: 边预测logits + :param pred2: label预测logits + :param target1: 真实边的标注 + :param target2: 真实类别的标注 + :param seq_len: 序列长度 + :return dict: 评估结果:: + + UAS: 不带label时, 边预测的准确率 + LAS: 同时预测边和label的准确率 + """ + + def __init__(self, pred1=None, pred2=None, + target1=None, target2=None, seq_len=None): + super().__init__() - self._init_param_map(arc_pred=arc_pred, label_pred=label_pred, - arc_true=arc_true, label_true=label_true, - seq_lens=seq_lens) + self._init_param_map(pred1=pred1, pred2=pred2, + target1=target1, target2=target2, + seq_len=seq_len) self.num_arc = 0 self.num_label = 0 self.num_sample = 0 - + def get_metric(self, reset=True): - res = {'UAS': self.num_arc*1.0 / self.num_sample, 'LAS': self.num_label*1.0 / self.num_sample} + res = {'UAS': self.num_arc * 1.0 / self.num_sample, 'LAS': self.num_label * 1.0 / self.num_sample} if reset: self.num_sample = self.num_label = self.num_arc = 0 return res - - def evaluate(self, arc_pred, label_pred, arc_true, label_true, seq_lens=None): + + def evaluate(self, pred1, pred2, target1, target2, seq_len=None): """Evaluate the performance of prediction. """ - if seq_lens is None: - seq_mask = arc_pred.new_ones(arc_pred.size(), dtype=torch.long) + if seq_len is None: + seq_mask = pred1.new_ones(pred1.size(), dtype=torch.long) else: - seq_mask = seq_lens_to_masks(seq_lens.long(), float=False).long() + seq_mask = seq_len_to_mask(seq_len.long()).long() # mask out tag - seq_mask[:,0] = 0 - head_pred_correct = (arc_pred == arc_true).long() * seq_mask - label_pred_correct = (label_pred == label_true).long() * head_pred_correct + seq_mask[:, 0] = 0 + head_pred_correct = (pred1 == target1).long() * seq_mask + label_pred_correct = (pred2 == target2).long() * head_pred_correct self.num_arc += head_pred_correct.sum().item() self.num_label += label_pred_correct.sum().item() self.num_sample += seq_mask.sum().item() diff --git a/fastNLP/models/char_language_model.py b/fastNLP/models/char_language_model.py deleted file mode 100644 index 5fbde3cc..00000000 --- a/fastNLP/models/char_language_model.py +++ /dev/null @@ -1,131 +0,0 @@ -import torch -import torch.nn as nn -import torch.nn.functional as F - -from fastNLP.modules.encoder.lstm import LSTM - - -class Highway(nn.Module): - """Highway network""" - - def __init__(self, input_size): - super(Highway, self).__init__() - self.fc1 = nn.Linear(input_size, input_size, bias=True) - self.fc2 = nn.Linear(input_size, input_size, bias=True) - - def forward(self, x): - t = F.sigmoid(self.fc1(x)) - return torch.mul(t, F.relu(self.fc2(x))) + torch.mul(1 - t, x) - - -class CharLM(nn.Module): - """CNN + highway network + LSTM - # Input: - 4D tensor with shape [batch_size, in_channel, height, width] - # Output: - 2D Tensor with shape [batch_size, vocab_size] - # Arguments: - char_emb_dim: the size of each character's attention - word_emb_dim: the size of each word's attention - vocab_size: num of unique words - num_char: num of characters - use_gpu: True or False - """ - - def __init__(self, char_emb_dim, word_emb_dim, - vocab_size, num_char): - super(CharLM, self).__init__() - self.char_emb_dim = char_emb_dim - self.word_emb_dim = word_emb_dim - self.vocab_size = vocab_size - - # char attention layer - self.char_embed = nn.Embedding(num_char, char_emb_dim) - - # convolutions of filters with different sizes - self.convolutions = [] - - # list of tuples: (the number of filter, width) - self.filter_num_width = [(25, 1), (50, 2), (75, 3), (100, 4), (125, 5), (150, 6)] - - for out_channel, filter_width in self.filter_num_width: - self.convolutions.append( - nn.Conv2d( - 1, # in_channel - out_channel, # out_channel - kernel_size=(char_emb_dim, filter_width), # (height, width) - bias=True - ) - ) - - self.highway_input_dim = sum([x for x, y in self.filter_num_width]) - - self.batch_norm = nn.BatchNorm1d(self.highway_input_dim, affine=False) - - # highway net - self.highway1 = Highway(self.highway_input_dim) - self.highway2 = Highway(self.highway_input_dim) - - # LSTM - self.lstm_num_layers = 2 - - self.lstm = LSTM(self.highway_input_dim, hidden_size=self.word_emb_dim, num_layers=self.lstm_num_layers, - dropout=0.5) - # output layer - self.dropout = nn.Dropout(p=0.5) - self.linear = nn.Linear(self.word_emb_dim, self.vocab_size) - - def forward(self, x): - # Input: Variable of Tensor with shape [num_seq, seq_len, max_word_len+2] - # Return: Variable of Tensor with shape [num_words, len(word_dict)] - lstm_batch_size = x.size()[0] - lstm_seq_len = x.size()[1] - - x = x.contiguous().view(-1, x.size()[2]) - # [num_seq*seq_len, max_word_len+2] - - x = self.char_embed(x) - # [num_seq*seq_len, max_word_len+2, char_emb_dim] - - x = torch.transpose(x.view(x.size()[0], 1, x.size()[1], -1), 2, 3) - # [num_seq*seq_len, 1, max_word_len+2, char_emb_dim] - - x = self.conv_layers(x) - # [num_seq*seq_len, total_num_filters] - - x = self.batch_norm(x) - # [num_seq*seq_len, total_num_filters] - - x = self.highway1(x) - x = self.highway2(x) - # [num_seq*seq_len, total_num_filters] - - x = x.contiguous().view(lstm_batch_size, lstm_seq_len, -1) - # [num_seq, seq_len, total_num_filters] - - x = self.lstm(x) - # [seq_len, num_seq, hidden_size] - - x = self.dropout(x) - # [seq_len, num_seq, hidden_size] - - x = x.contiguous().view(lstm_batch_size * lstm_seq_len, -1) - # [num_seq*seq_len, hidden_size] - - x = self.linear(x) - # [num_seq*seq_len, vocab_size] - return x - - def conv_layers(self, x): - chosen_list = list() - for conv in self.convolutions: - feature_map = F.tanh(conv(x)) - # (batch_size, out_channel, 1, max_word_len-width+1) - chosen = torch.max(feature_map, 3)[0] - # (batch_size, out_channel, 1) - chosen = chosen.squeeze() - # (batch_size, out_channel) - chosen_list.append(chosen) - - # (batch_size, total_num_filers) - return torch.cat(chosen_list, 1) diff --git a/fastNLP/models/cnn_text_classification.py b/fastNLP/models/cnn_text_classification.py index f3898c00..3a71a80a 100644 --- a/fastNLP/models/cnn_text_classification.py +++ b/fastNLP/models/cnn_text_classification.py @@ -1,57 +1,68 @@ -# python: 3.6 -# encoding: utf-8 +__all__ = [ + "CNNText" +] import torch import torch.nn as nn -# import torch.nn.functional as F -import fastNLP.modules.encoder as encoder +from ..core.const import Const as C +from ..modules import encoder class CNNText(torch.nn.Module): """ - Text classification model by character CNN, the implementation of paper - 'Yoon Kim. 2014. Convolution Neural Networks for Sentence - Classification.' - """ + 别名：:class:`fastNLP.models.CNNText` :class:`fastNLP.models.cnn_text_classification.CNNText` - def __init__(self, embed_num, - embed_dim, + 使用CNN进行文本分类的模型 + 'Yoon Kim. 2014. Convolution Neural Networks for Sentence Classification.' + + :param tuple(int,int),torch.FloatTensor,nn.Embedding,numpy.ndarray init_embed: Embedding的大小(传入tuple(int, int), + 第一个int为vocab_zie, 第二个int为embed_dim); 如果为Tensor, Embedding, ndarray等则直接使用该值初始化Embedding + :param int num_classes: 一共有多少类 + :param int,tuple(int) out_channels: 输出channel的数量。如果为list，则需要与kernel_sizes的数量保持一致 + :param int,tuple(int) kernel_sizes: 输出channel的kernel大小。 + :param int padding: 对句子前后的pad的大小, 用0填充。 + :param float dropout: Dropout的大小 + """ + + def __init__(self, init_embed, num_classes, kernel_nums=(3, 4, 5), kernel_sizes=(3, 4, 5), padding=0, dropout=0.5): super(CNNText, self).__init__() - + # no support for pre-trained embedding currently - self.embed = encoder.Embedding(embed_num, embed_dim) + self.embed = encoder.Embedding(init_embed) self.conv_pool = encoder.ConvMaxpool( - in_channels=embed_dim, + in_channels=self.embed.embedding_dim, out_channels=kernel_nums, kernel_sizes=kernel_sizes, padding=padding) self.dropout = nn.Dropout(dropout) - self.fc = encoder.Linear(sum(kernel_nums), num_classes) - - def forward(self, word_seq): + self.fc = nn.Linear(sum(kernel_nums), num_classes) + + def forward(self, words, seq_len=None): """ - :param word_seq: torch.LongTensor, [batch_size, seq_len] + :param torch.LongTensor words: [batch_size, seq_len]，句子中word的index + :param torch.LongTensor seq_len: [batch,] 每个句子的长度 :return output: dict of torch.LongTensor, [batch_size, num_classes] """ - x = self.embed(word_seq) # [N,L] -> [N,L,C] + x = self.embed(words) # [N,L] -> [N,L,C] x = self.conv_pool(x) # [N,L,C] -> [N,C] x = self.dropout(x) x = self.fc(x) # [N,C] -> [N, N_class] - return {'pred': x} - - def predict(self, word_seq): + return {C.OUTPUT: x} + + def predict(self, words, seq_len=None): """ + :param torch.LongTensor words: [batch_size, seq_len]，句子中word的index + :param torch.LongTensor seq_len: [batch,] 每个句子的长度 - :param word_seq: torch.LongTensor, [batch_size, seq_len] - :return predict: dict of torch.LongTensor, [batch_size, seq_len] + :return predict: dict of torch.LongTensor, [batch_size, ] """ - output = self(word_seq) - _, predict = output['pred'].max(dim=1) - return {'pred': predict} + output = self(words, seq_len) + _, predict = output[C.OUTPUT].max(dim=1) + return {C.OUTPUT: predict} diff --git a/fastNLP/models/enas_controller.py b/fastNLP/models/enas_controller.py new file mode 100644 index 00000000..e83c6b51 --- /dev/null +++ b/fastNLP/models/enas_controller.py @@ -0,0 +1,223 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch +"""A module with NAS controller-related code.""" +import collections +import os + +import torch +import torch.nn.functional as F + +from . import enas_utils as utils +from .enas_utils import Node + + +def _construct_dags(prev_nodes, activations, func_names, num_blocks): + """Constructs a set of DAGs based on the actions, i.e., previous nodes and + activation functions, sampled from the controller/policy pi. + + Args: + prev_nodes: Previous node actions from the policy. + activations: Activations sampled from the policy. + func_names: Mapping from activation function names to functions. + num_blocks: Number of blocks in the target RNN cell. + + Returns: + A list of DAGs defined by the inputs. + + RNN cell DAGs are represented in the following way: + + 1. Each element (node) in a DAG is a list of `Node`s. + + 2. The `Node`s in the list dag[i] correspond to the subsequent nodes + that take the output from node i as their own input. + + 3. dag[-1] is the node that takes input from x^{(t)} and h^{(t - 1)}. + dag[-1] always feeds dag[0]. + dag[-1] acts as if `w_xc`, `w_hc`, `w_xh` and `w_hh` are its + weights. + + 4. dag[N - 1] is the node that produces the hidden state passed to + the next timestep. dag[N - 1] is also always a leaf node, and therefore + is always averaged with the other leaf nodes and fed to the output + decoder. + """ + dags = [] + for nodes, func_ids in zip(prev_nodes, activations): + dag = collections.defaultdict(list) + + # add first node + dag[-1] = [Node(0, func_names[func_ids[0]])] + dag[-2] = [Node(0, func_names[func_ids[0]])] + + # add following nodes + for jdx, (idx, func_id) in enumerate(zip(nodes, func_ids[1:])): + dag[utils.to_item(idx)].append(Node(jdx + 1, func_names[func_id])) + + leaf_nodes = set(range(num_blocks)) - dag.keys() + + # merge with avg + for idx in leaf_nodes: + dag[idx] = [Node(num_blocks, 'avg')] + + # This is actually y^{(t)}. h^{(t)} is node N - 1 in + # the graph, where N Is the number of nodes. I.e., h^{(t)} takes + # only one other node as its input. + # last h[t] node + last_node = Node(num_blocks + 1, 'h[t]') + dag[num_blocks] = [last_node] + dags.append(dag) + + return dags + + +class Controller(torch.nn.Module): + """Based on + https://github.com/pytorch/examples/blob/master/word_language_model/model.py + + RL controllers do not necessarily have much to do with + language models. + + Base the controller RNN on the GRU from: + https://github.com/ikostrikov/pytorch-a2c-ppo-acktr/blob/master/model.py + """ + def __init__(self, num_blocks=4, controller_hid=100, cuda=False): + torch.nn.Module.__init__(self) + + # `num_tokens` here is just the activation function + # for every even step, + self.shared_rnn_activations = ['tanh', 'ReLU', 'identity', 'sigmoid'] + self.num_tokens = [len(self.shared_rnn_activations)] + self.controller_hid = controller_hid + self.use_cuda = cuda + self.num_blocks = num_blocks + for idx in range(num_blocks): + self.num_tokens += [idx + 1, len(self.shared_rnn_activations)] + self.func_names = self.shared_rnn_activations + + num_total_tokens = sum(self.num_tokens) + + self.encoder = torch.nn.Embedding(num_total_tokens, + controller_hid) + self.lstm = torch.nn.LSTMCell(controller_hid, controller_hid) + + # Perhaps these weights in the decoder should be + # shared? At least for the activation functions, which all have the + # same size. + self.decoders = [] + for idx, size in enumerate(self.num_tokens): + decoder = torch.nn.Linear(controller_hid, size) + self.decoders.append(decoder) + + self._decoders = torch.nn.ModuleList(self.decoders) + + self.reset_parameters() + self.static_init_hidden = utils.keydefaultdict(self.init_hidden) + + def _get_default_hidden(key): + return utils.get_variable( + torch.zeros(key, self.controller_hid), + self.use_cuda, + requires_grad=False) + + self.static_inputs = utils.keydefaultdict(_get_default_hidden) + + def reset_parameters(self): + init_range = 0.1 + for param in self.parameters(): + param.data.uniform_(-init_range, init_range) + for decoder in self.decoders: + decoder.bias.data.fill_(0) + + def forward(self, # pylint:disable=arguments-differ + inputs, + hidden, + block_idx, + is_embed): + if not is_embed: + embed = self.encoder(inputs) + else: + embed = inputs + + hx, cx = self.lstm(embed, hidden) + logits = self.decoders[block_idx](hx) + + logits /= 5.0 + + # # exploration + # if self.args.mode == 'train': + # logits = (2.5 * F.tanh(logits)) + + return logits, (hx, cx) + + def sample(self, batch_size=1, with_details=False, save_dir=None): + """Samples a set of `args.num_blocks` many computational nodes from the + controller, where each node is made up of an activation function, and + each node except the last also includes a previous node. + """ + if batch_size < 1: + raise Exception(f'Wrong batch_size: {batch_size} < 1') + + # [B, L, H] + inputs = self.static_inputs[batch_size] + hidden = self.static_init_hidden[batch_size] + + activations = [] + entropies = [] + log_probs = [] + prev_nodes = [] + # The RNN controller alternately outputs an activation, + # followed by a previous node, for each block except the last one, + # which only gets an activation function. The last node is the output + # node, and its previous node is the average of all leaf nodes. + for block_idx in range(2*(self.num_blocks - 1) + 1): + logits, hidden = self.forward(inputs, + hidden, + block_idx, + is_embed=(block_idx == 0)) + + probs = F.softmax(logits, dim=-1) + log_prob = F.log_softmax(logits, dim=-1) + # .mean() for entropy? + entropy = -(log_prob * probs).sum(1, keepdim=False) + + action = probs.multinomial(num_samples=1).data + selected_log_prob = log_prob.gather( + 1, utils.get_variable(action, requires_grad=False)) + + # why the [:, 0] here? Should it be .squeeze(), or + # .view()? Same below with `action`. + entropies.append(entropy) + log_probs.append(selected_log_prob[:, 0]) + + # 0: function, 1: previous node + mode = block_idx % 2 + inputs = utils.get_variable( + action[:, 0] + sum(self.num_tokens[:mode]), + requires_grad=False) + + if mode == 0: + activations.append(action[:, 0]) + elif mode == 1: + prev_nodes.append(action[:, 0]) + + prev_nodes = torch.stack(prev_nodes).transpose(0, 1) + activations = torch.stack(activations).transpose(0, 1) + + dags = _construct_dags(prev_nodes, + activations, + self.func_names, + self.num_blocks) + + if save_dir is not None: + for idx, dag in enumerate(dags): + utils.draw_network(dag, + os.path.join(save_dir, f'graph{idx}.png')) + + if with_details: + return dags, torch.cat(log_probs), torch.cat(entropies) + + return dags + + def init_hidden(self, batch_size): + zeros = torch.zeros(batch_size, self.controller_hid) + return (utils.get_variable(zeros, self.use_cuda, requires_grad=False), + utils.get_variable(zeros.clone(), self.use_cuda, requires_grad=False)) diff --git a/fastNLP/models/enas_model.py b/fastNLP/models/enas_model.py new file mode 100644 index 00000000..b6b683c0 --- /dev/null +++ b/fastNLP/models/enas_model.py @@ -0,0 +1,390 @@ +""" +Module containing the shared RNN model. +Code Modified from https://github.com/carpedm20/ENAS-pytorch +""" +import collections + +import numpy as np +import torch +import torch.nn as nn +import torch.nn.functional as F +from torch.autograd import Variable + +from . import enas_utils as utils +from .base_model import BaseModel + + +def _get_dropped_weights(w_raw, dropout_p, is_training): + """Drops out weights to implement DropConnect. + + Args: + w_raw: Full, pre-dropout, weights to be dropped out. + dropout_p: Proportion of weights to drop out. + is_training: True iff _shared_ model is training. + + Returns: + The dropped weights. + + Why does torch.nn.functional.dropout() return: + 1. `torch.autograd.Variable()` on the training loop + 2. `torch.nn.Parameter()` on the controller or eval loop, when + training = False... + + Even though the call to `_setweights` in the Smerity repo's + `weight_drop.py` does not have this behaviour, and `F.dropout` always + returns `torch.autograd.Variable` there, even when `training=False`? + + The above TODO is the reason for the hacky check for `torch.nn.Parameter`. + """ + dropped_w = F.dropout(w_raw, p=dropout_p, training=is_training) + + if isinstance(dropped_w, torch.nn.Parameter): + dropped_w = dropped_w.clone() + + return dropped_w + + +class EmbeddingDropout(torch.nn.Embedding): + """Class for dropping out embeddings by zero'ing out parameters in the + embedding matrix. + + This is equivalent to dropping out particular words, e.g., in the sentence + 'the quick brown fox jumps over the lazy dog', dropping out 'the' would + lead to the sentence '### quick brown fox jumps over ### lazy dog' (in the + embedding vector space). + + See 'A Theoretically Grounded Application of Dropout in Recurrent Neural + Networks', (Gal and Ghahramani, 2016). + """ + + def __init__(self, + num_embeddings, + embedding_dim, + max_norm=None, + norm_type=2, + scale_grad_by_freq=False, + sparse=False, + dropout=0.1, + scale=None): + """Embedding constructor. + + Args: + dropout: Dropout probability. + scale: Used to scale parameters of embedding weight matrix that are + not dropped out. Note that this is _in addition_ to the + `1/(1 - dropout)` scaling. + + See `torch.nn.Embedding` for remaining arguments. + """ + torch.nn.Embedding.__init__(self, + num_embeddings=num_embeddings, + embedding_dim=embedding_dim, + max_norm=max_norm, + norm_type=norm_type, + scale_grad_by_freq=scale_grad_by_freq, + sparse=sparse) + self.dropout = dropout + assert (dropout >= 0.0) and (dropout < 1.0), ('Dropout must be >= 0.0 ' + 'and < 1.0') + self.scale = scale + + def forward(self, inputs): # pylint:disable=arguments-differ + """Embeds `inputs` with the dropped out embedding weight matrix.""" + if self.training: + dropout = self.dropout + else: + dropout = 0 + + if dropout: + mask = self.weight.data.new(self.weight.size(0), 1) + mask.bernoulli_(1 - dropout) + mask = mask.expand_as(self.weight) + mask = mask / (1 - dropout) + masked_weight = self.weight * Variable(mask) + else: + masked_weight = self.weight + if self.scale and self.scale != 1: + masked_weight = masked_weight * self.scale + + return F.embedding(inputs, + masked_weight, + max_norm=self.max_norm, + norm_type=self.norm_type, + scale_grad_by_freq=self.scale_grad_by_freq, + sparse=self.sparse) + + +class LockedDropout(nn.Module): + # code from https://github.com/salesforce/awd-lstm-lm/blob/master/locked_dropout.py + def __init__(self): + super().__init__() + + def forward(self, x, dropout=0.5): + if not self.training or not dropout: + return x + m = x.data.new(1, x.size(1), x.size(2)).bernoulli_(1 - dropout) + mask = Variable(m, requires_grad=False) / (1 - dropout) + mask = mask.expand_as(x) + return mask * x + + +class ENASModel(BaseModel): + """Shared RNN model.""" + + def __init__(self, embed_num, num_classes, num_blocks=4, cuda=False, shared_hid=1000, shared_embed=1000): + super(ENASModel, self).__init__() + + self.use_cuda = cuda + + self.shared_hid = shared_hid + self.num_blocks = num_blocks + self.decoder = nn.Linear(self.shared_hid, num_classes) + self.encoder = EmbeddingDropout(embed_num, + shared_embed, + dropout=0.1) + self.lockdrop = LockedDropout() + self.dag = None + + # Tie weights + # self.decoder.weight = self.encoder.weight + + # Since W^{x, c} and W^{h, c} are always summed, there + # is no point duplicating their bias offset parameter. Likewise for + # W^{x, h} and W^{h, h}. + self.w_xc = nn.Linear(shared_embed, self.shared_hid) + self.w_xh = nn.Linear(shared_embed, self.shared_hid) + + # The raw weights are stored here because the hidden-to-hidden weights + # are weight dropped on the forward pass. + self.w_hc_raw = torch.nn.Parameter( + torch.Tensor(self.shared_hid, self.shared_hid)) + self.w_hh_raw = torch.nn.Parameter( + torch.Tensor(self.shared_hid, self.shared_hid)) + self.w_hc = None + self.w_hh = None + + self.w_h = collections.defaultdict(dict) + self.w_c = collections.defaultdict(dict) + + for idx in range(self.num_blocks): + for jdx in range(idx + 1, self.num_blocks): + self.w_h[idx][jdx] = nn.Linear(self.shared_hid, + self.shared_hid, + bias=False) + self.w_c[idx][jdx] = nn.Linear(self.shared_hid, + self.shared_hid, + bias=False) + + self._w_h = nn.ModuleList([self.w_h[idx][jdx] + for idx in self.w_h + for jdx in self.w_h[idx]]) + self._w_c = nn.ModuleList([self.w_c[idx][jdx] + for idx in self.w_c + for jdx in self.w_c[idx]]) + + self.batch_norm = None + # if args.mode == 'train': + # self.batch_norm = nn.BatchNorm1d(self.shared_hid) + # else: + # self.batch_norm = None + + self.reset_parameters() + self.static_init_hidden = utils.keydefaultdict(self.init_hidden) + + def setDAG(self, dag): + if self.dag is None: + self.dag = dag + + def forward(self, word_seq, hidden=None): + inputs = torch.transpose(word_seq, 0, 1) + + time_steps = inputs.size(0) + batch_size = inputs.size(1) + + self.w_hh = _get_dropped_weights(self.w_hh_raw, + 0.5, + self.training) + self.w_hc = _get_dropped_weights(self.w_hc_raw, + 0.5, + self.training) + + # hidden = self.static_init_hidden[batch_size] if hidden is None else hidden + hidden = self.static_init_hidden[batch_size] + + embed = self.encoder(inputs) + + embed = self.lockdrop(embed, 0.65 if self.training else 0) + + # The norm of hidden states are clipped here because + # otherwise ENAS is especially prone to exploding activations on the + # forward pass. This could probably be fixed in a more elegant way, but + # it might be exposing a weakness in the ENAS algorithm as currently + # proposed. + # + # For more details, see + # https://github.com/carpedm20/ENAS-pytorch/issues/6 + clipped_num = 0 + max_clipped_norm = 0 + h1tohT = [] + logits = [] + for step in range(time_steps): + x_t = embed[step] + logit, hidden = self.cell(x_t, hidden, self.dag) + + hidden_norms = hidden.norm(dim=-1) + max_norm = 25.0 + if hidden_norms.data.max() > max_norm: + # Just directly use the torch slice operations + # in PyTorch v0.4. + # + # This workaround for PyTorch v0.3.1 does everything in numpy, + # because the PyTorch slicing and slice assignment is too + # flaky. + hidden_norms = hidden_norms.data.cpu().numpy() + + clipped_num += 1 + if hidden_norms.max() > max_clipped_norm: + max_clipped_norm = hidden_norms.max() + + clip_select = hidden_norms > max_norm + clip_norms = hidden_norms[clip_select] + + mask = np.ones(hidden.size()) + normalizer = max_norm / clip_norms + normalizer = normalizer[:, np.newaxis] + + mask[clip_select] = normalizer + + if self.use_cuda: + hidden *= torch.autograd.Variable( + torch.FloatTensor(mask).cuda(), requires_grad=False) + else: + hidden *= torch.autograd.Variable( + torch.FloatTensor(mask), requires_grad=False) + logits.append(logit) + h1tohT.append(hidden) + + h1tohT = torch.stack(h1tohT) + output = torch.stack(logits) + raw_output = output + + output = self.lockdrop(output, 0.4 if self.training else 0) + + # Pooling + output = torch.mean(output, 0) + + decoded = self.decoder(output) + + extra_out = {'dropped': decoded, + 'hiddens': h1tohT, + 'raw': raw_output} + return {'pred': decoded, 'hidden': hidden, 'extra_out': extra_out} + + def cell(self, x, h_prev, dag): + """Computes a single pass through the discovered RNN cell.""" + c = {} + h = {} + f = {} + + f[0] = self.get_f(dag[-1][0].name) + c[0] = torch.sigmoid(self.w_xc(x) + F.linear(h_prev, self.w_hc, None)) + h[0] = (c[0] * f[0](self.w_xh(x) + F.linear(h_prev, self.w_hh, None)) + + (1 - c[0]) * h_prev) + + leaf_node_ids = [] + q = collections.deque() + q.append(0) + + # Computes connections from the parent nodes `node_id` + # to their child nodes `next_id` recursively, skipping leaf nodes. A + # leaf node is a node whose id == `self.num_blocks`. + # + # Connections between parent i and child j should be computed as + # h_j = c_j*f_{ij}{(W^h_{ij}*h_i)} + (1 - c_j)*h_i, + # where c_j = \sigmoid{(W^c_{ij}*h_i)} + # + # See Training details from Section 3.1 of the paper. + # + # The following algorithm does a breadth-first (since `q.popleft()` is + # used) search over the nodes and computes all the hidden states. + while True: + if len(q) == 0: + break + + node_id = q.popleft() + nodes = dag[node_id] + + for next_node in nodes: + next_id = next_node.id + if next_id == self.num_blocks: + leaf_node_ids.append(node_id) + assert len(nodes) == 1, ('parent of leaf node should have ' + 'only one child') + continue + + w_h = self.w_h[node_id][next_id] + w_c = self.w_c[node_id][next_id] + + f[next_id] = self.get_f(next_node.name) + c[next_id] = torch.sigmoid(w_c(h[node_id])) + h[next_id] = (c[next_id] * f[next_id](w_h(h[node_id])) + + (1 - c[next_id]) * h[node_id]) + + q.append(next_id) + + # Instead of averaging loose ends, perhaps there should + # be a set of separate unshared weights for each "loose" connection + # between each node in a cell and the output. + # + # As it stands, all weights W^h_{ij} are doing double duty by + # connecting both from i to j, as well as from i to the output. + + # average all the loose ends + leaf_nodes = [h[node_id] for node_id in leaf_node_ids] + output = torch.mean(torch.stack(leaf_nodes, 2), -1) + + # stabilizing the Updates of omega + if self.batch_norm is not None: + output = self.batch_norm(output) + + return output, h[self.num_blocks - 1] + + def init_hidden(self, batch_size): + zeros = torch.zeros(batch_size, self.shared_hid) + return utils.get_variable(zeros, self.use_cuda, requires_grad=False) + + def get_f(self, name): + name = name.lower() + if name == 'relu': + f = torch.relu + elif name == 'tanh': + f = torch.tanh + elif name == 'identity': + f = lambda x: x + elif name == 'sigmoid': + f = torch.sigmoid + return f + + @property + def num_parameters(self): + def size(p): + return np.prod(p.size()) + + return sum([size(param) for param in self.parameters()]) + + def reset_parameters(self): + init_range = 0.025 + # init_range = 0.025 if self.args.mode == 'train' else 0.04 + for param in self.parameters(): + param.data.uniform_(-init_range, init_range) + self.decoder.bias.data.fill_(0) + + def predict(self, word_seq): + """ + + :param word_seq: torch.LongTensor, [batch_size, seq_len] + :return predict: dict of torch.LongTensor, [batch_size, seq_len] + """ + output = self(word_seq) + _, predict = output['pred'].max(dim=1) + return {'pred': predict} diff --git a/fastNLP/models/enas_trainer.py b/fastNLP/models/enas_trainer.py new file mode 100644 index 00000000..ef596b03 --- /dev/null +++ b/fastNLP/models/enas_trainer.py @@ -0,0 +1,380 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch +import math +import numpy as np +import time +import torch + +from datetime import datetime, timedelta + +from torch.optim import Adam + +try: + from tqdm.auto import tqdm +except: + from ..core.utils import _pseudo_tqdm as tqdm + +from ..core.trainer import Trainer +from ..core.batch import Batch +from ..core.callback import CallbackManager, CallbackException +from ..core.dataset import DataSet +from ..core.utils import _move_dict_value_to_device +from . import enas_utils as utils +from ..core.utils import _build_args + + +def _get_no_grad_ctx_mgr(): + """Returns a the `torch.no_grad` context manager for PyTorch version >= + 0.4, or a no-op context manager otherwise. + """ + return torch.no_grad() + + +class ENASTrainer(Trainer): + """A class to wrap training code.""" + + def __init__(self, train_data, model, controller, **kwargs): + """Constructor for training algorithm. + :param DataSet train_data: the training data + :param torch.nn.modules.module model: a PyTorch model + :param torch.nn.modules.module controller: a PyTorch model + """ + self.final_epochs = kwargs['final_epochs'] + kwargs.pop('final_epochs') + super(ENASTrainer, self).__init__(train_data, model, **kwargs) + self.controller_step = 0 + self.shared_step = 0 + self.max_length = 35 + + self.shared = model + self.controller = controller + + self.shared_optim = Adam( + self.shared.parameters(), + lr=20.0, + weight_decay=1e-7) + + self.controller_optim = Adam( + self.controller.parameters(), + lr=3.5e-4) + + def train(self, load_best_model=True): + """ + :param bool load_best_model: 该参数只有在初始化提供了dev_data的情况下有效，如果True, trainer将在返回之前重新加载dev表现 + 最好的模型参数。 + :return results: 返回一个字典类型的数据, + 内含以下内容:: + + seconds: float, 表示训练时长 + 以下三个内容只有在提供了dev_data的情况下会有。 + best_eval: Dict of Dict, 表示evaluation的结果 + best_epoch: int，在第几个epoch取得的最佳值 + best_step: int, 在第几个step(batch)更新取得的最佳值 + + """ + results = {} + if self.n_epochs <= 0: + print(f"training epoch is {self.n_epochs}, nothing was done.") + results['seconds'] = 0. + return results + try: + if torch.cuda.is_available() and "cuda" in self.device: + self.model = self.model.cuda() + self._model_device = self.model.parameters().__next__().device + self._mode(self.model, is_test=False) + + self.start_time = str(datetime.now().strftime('%Y-%m-%d-%H-%M-%S')) + start_time = time.time() + print("training epochs started " + self.start_time, flush=True) + + try: + self.callback_manager.on_train_begin() + self._train() + self.callback_manager.on_train_end() + except (CallbackException, KeyboardInterrupt) as e: + self.callback_manager.on_exception(e) + + if self.dev_data is not None: + print( + "\nIn Epoch:{}/Step:{}, got best dev performance:".format(self.best_dev_epoch, self.best_dev_step) + + self.tester._format_eval_results(self.best_dev_perf), ) + results['best_eval'] = self.best_dev_perf + results['best_epoch'] = self.best_dev_epoch + results['best_step'] = self.best_dev_step + if load_best_model: + model_name = "best_" + "_".join([self.model.__class__.__name__, self.metric_key, self.start_time]) + load_succeed = self._load_model(self.model, model_name) + if load_succeed: + print("Reloaded the best model.") + else: + print("Fail to reload best model.") + finally: + pass + results['seconds'] = round(time.time() - start_time, 2) + + return results + + def _train(self): + if not self.use_tqdm: + from fastNLP.core.utils import _pseudo_tqdm as inner_tqdm + else: + inner_tqdm = tqdm + self.step = 0 + start = time.time() + total_steps = (len(self.train_data) // self.batch_size + int( + len(self.train_data) % self.batch_size != 0)) * self.n_epochs + with inner_tqdm(total=total_steps, postfix='loss:{0:<6.5f}', leave=False, dynamic_ncols=True) as pbar: + avg_loss = 0 + data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + for epoch in range(1, self.n_epochs + 1): + pbar.set_description_str(desc="Epoch {}/{}".format(epoch, self.n_epochs)) + last_stage = (epoch > self.n_epochs + 1 - self.final_epochs) + if epoch == self.n_epochs + 1 - self.final_epochs: + print('Entering the final stage. (Only train the selected structure)') + # early stopping + self.callback_manager.on_epoch_begin() + + # 1. Training the shared parameters omega of the child models + self.train_shared(pbar) + + # 2. Training the controller parameters theta + if not last_stage: + self.train_controller() + + if ((self.validate_every > 0 and self.step % self.validate_every == 0) or + (self.validate_every < 0 and self.step % len(data_iterator) == 0)) \ + and self.dev_data is not None: + if not last_stage: + self.derive() + eval_res = self._do_validation(epoch=epoch, step=self.step) + eval_str = "Evaluation at Epoch {}/{}. Step:{}/{}. ".format(epoch, self.n_epochs, self.step, + total_steps) + \ + self.tester._format_eval_results(eval_res) + pbar.write(eval_str) + + # lr decay; early stopping + self.callback_manager.on_epoch_end() + # =============== epochs end =================== # + pbar.close() + # ============ tqdm end ============== # + + def get_loss(self, inputs, targets, hidden, dags): + """Computes the loss for the same batch for M models. + + This amounts to an estimate of the loss, which is turned into an + estimate for the gradients of the shared model. + """ + if not isinstance(dags, list): + dags = [dags] + + loss = 0 + for dag in dags: + self.shared.setDAG(dag) + inputs = _build_args(self.shared.forward, **inputs) + inputs['hidden'] = hidden + result = self.shared(**inputs) + output, hidden, extra_out = result['pred'], result['hidden'], result['extra_out'] + + self.callback_manager.on_loss_begin(targets, result) + sample_loss = self._compute_loss(result, targets) + loss += sample_loss + + assert len(dags) == 1, 'there are multiple `hidden` for multple `dags`' + return loss, hidden, extra_out + + def train_shared(self, pbar=None, max_step=None, dag=None): + """Train the language model for 400 steps of minibatches of 64 + examples. + + Args: + max_step: Used to run extra training steps as a warm-up. + dag: If not None, is used instead of calling sample(). + + BPTT is truncated at 35 timesteps. + + For each weight update, gradients are estimated by sampling M models + from the fixed controller policy, and averaging their gradients + computed on a batch of training data. + """ + model = self.shared + model.train() + self.controller.eval() + + hidden = self.shared.init_hidden(self.batch_size) + + abs_max_grad = 0 + abs_max_hidden_norm = 0 + step = 0 + raw_total_loss = 0 + total_loss = 0 + train_idx = 0 + avg_loss = 0 + data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + + for batch_x, batch_y in data_iterator: + _move_dict_value_to_device(batch_x, batch_y, device=self._model_device) + indices = data_iterator.get_batch_indices() + # negative sampling; replace unknown; re-weight batch_y + self.callback_manager.on_batch_begin(batch_x, batch_y, indices) + # prediction = self._data_forward(self.model, batch_x) + + dags = self.controller.sample(1) + inputs, targets = batch_x, batch_y + # self.callback_manager.on_loss_begin(batch_y, prediction) + loss, hidden, extra_out = self.get_loss(inputs, + targets, + hidden, + dags) + hidden.detach_() + + avg_loss += loss.item() + + # Is loss NaN or inf? requires_grad = False + self.callback_manager.on_backward_begin(loss) + self._grad_backward(loss) + self.callback_manager.on_backward_end() + + self._update() + self.callback_manager.on_step_end() + + if (self.step + 1) % self.print_every == 0: + if self.use_tqdm: + print_output = "loss:{0:<6.5f}".format(avg_loss / self.print_every) + pbar.update(self.print_every) + else: + end = time.time() + diff = timedelta(seconds=round(end - start)) + print_output = "[epoch: {:>3} step: {:>4}] train loss: {:>4.6} time: {}".format( + epoch, self.step, avg_loss, diff) + pbar.set_postfix_str(print_output) + avg_loss = 0 + self.step += 1 + step += 1 + self.shared_step += 1 + self.callback_manager.on_batch_end() + # ================= mini-batch end ==================== # + + def get_reward(self, dag, entropies, hidden, valid_idx=0): + """Computes the perplexity of a single sampled model on a minibatch of + validation data. + """ + if not isinstance(entropies, np.ndarray): + entropies = entropies.data.cpu().numpy() + + data_iterator = Batch(self.dev_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + + for inputs, targets in data_iterator: + valid_loss, hidden, _ = self.get_loss(inputs, targets, hidden, dag) + valid_loss = utils.to_item(valid_loss.data) + + valid_ppl = math.exp(valid_loss) + + R = 80 / valid_ppl + + rewards = R + 1e-4 * entropies + + return rewards, hidden + + def train_controller(self): + """Fixes the shared parameters and updates the controller parameters. + + The controller is updated with a score function gradient estimator + (i.e., REINFORCE), with the reward being c/valid_ppl, where valid_ppl + is computed on a minibatch of validation data. + + A moving average baseline is used. + + The controller is trained for 2000 steps per epoch (i.e., + first (Train Shared) phase -> second (Train Controller) phase). + """ + model = self.controller + model.train() + # Why can't we call shared.eval() here? Leads to loss + # being uniformly zero for the controller. + # self.shared.eval() + + avg_reward_base = None + baseline = None + adv_history = [] + entropy_history = [] + reward_history = [] + + hidden = self.shared.init_hidden(self.batch_size) + total_loss = 0 + valid_idx = 0 + for step in range(20): + # sample models + dags, log_probs, entropies = self.controller.sample( + with_details=True) + + # calculate reward + np_entropies = entropies.data.cpu().numpy() + # No gradients should be backpropagated to the + # shared model during controller training, obviously. + with _get_no_grad_ctx_mgr(): + rewards, hidden = self.get_reward(dags, + np_entropies, + hidden, + valid_idx) + + reward_history.extend(rewards) + entropy_history.extend(np_entropies) + + # moving average baseline + if baseline is None: + baseline = rewards + else: + decay = 0.95 + baseline = decay * baseline + (1 - decay) * rewards + + adv = rewards - baseline + adv_history.extend(adv) + + # policy loss + loss = -log_probs * utils.get_variable(adv, + 'cuda' in self.device, + requires_grad=False) + + loss = loss.sum() # or loss.mean() + + # update + self.controller_optim.zero_grad() + loss.backward() + + self.controller_optim.step() + + total_loss += utils.to_item(loss.data) + + if ((step % 50) == 0) and (step > 0): + reward_history, adv_history, entropy_history = [], [], [] + total_loss = 0 + + self.controller_step += 1 + # prev_valid_idx = valid_idx + # valid_idx = ((valid_idx + self.max_length) % + # (self.valid_data.size(0) - 1)) + # # Whenever we wrap around to the beginning of the + # # validation data, we reset the hidden states. + # if prev_valid_idx > valid_idx: + # hidden = self.shared.init_hidden(self.batch_size) + + def derive(self, sample_num=10, valid_idx=0): + """We are always deriving based on the very first batch + of validation data? This seems wrong... + """ + hidden = self.shared.init_hidden(self.batch_size) + + dags, _, entropies = self.controller.sample(sample_num, + with_details=True) + + max_R = 0 + best_dag = None + for dag in dags: + R, _ = self.get_reward(dag, entropies, hidden, valid_idx) + if R.max() > max_R: + max_R = R.max() + best_dag = dag + + self.model.setDAG(best_dag) diff --git a/fastNLP/models/enas_utils.py b/fastNLP/models/enas_utils.py new file mode 100644 index 00000000..4e402a9a --- /dev/null +++ b/fastNLP/models/enas_utils.py @@ -0,0 +1,54 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch + +from collections import defaultdict +import collections + +import numpy as np +import torch +from torch.autograd import Variable + + +def detach(h): + if type(h) == Variable: + return Variable(h.data) + else: + return tuple(detach(v) for v in h) + + +def get_variable(inputs, cuda=False, **kwargs): + if type(inputs) in [list, np.ndarray]: + inputs = torch.Tensor(inputs) + if cuda: + out = Variable(inputs.cuda(), **kwargs) + else: + out = Variable(inputs, **kwargs) + return out + + +def update_lr(optimizer, lr): + for param_group in optimizer.param_groups: + param_group['lr'] = lr + + +Node = collections.namedtuple('Node', ['id', 'name']) + + +class keydefaultdict(defaultdict): + def __missing__(self, key): + if self.default_factory is None: + raise KeyError(key) + else: + ret = self[key] = self.default_factory(key) + return ret + + +def to_item(x): + """Converts x, possibly scalar and possibly tensor, to a Python scalar.""" + if isinstance(x, (float, int)): + return x + + if float(torch.__version__[0:3]) < 0.4: + assert (x.dim() == 1) and (len(x) == 1) + return x[0] + + return x.item() diff --git a/fastNLP/models/sequence_labeling.py b/fastNLP/models/sequence_labeling.py new file mode 100644 index 00000000..8e6a5db1 --- /dev/null +++ b/fastNLP/models/sequence_labeling.py @@ -0,0 +1,233 @@ +""" + 本模块实现了两种序列标注模型 +""" +__all__ = [ + "SeqLabeling", + "AdvSeqLabel" +] + +import torch +import torch.nn as nn + +from .base_model import BaseModel +from ..modules import decoder, encoder +from ..modules.decoder.crf import allowed_transitions +from ..core.utils import seq_len_to_mask +from ..core.const import Const as C + + +class SeqLabeling(BaseModel): + """ + 别名：:class:`fastNLP.models.SeqLabeling` :class:`fastNLP.models.sequence_labeling.SeqLabeling` + + 一个基础的Sequence labeling的模型。 + 用于做sequence labeling的基础类。结构包含一层Embedding，一层LSTM(单向，一层)，一层FC，以及一层CRF。 + + :param tuple(int,int),torch.FloatTensor,nn.Embedding,numpy.ndarray init_embed: Embedding的大小(传入tuple(int, int), + 第一个int为vocab_zie, 第二个int为embed_dim); 如果为Tensor, Embedding, ndarray等则直接使用该值初始化Embedding + :param int hidden_size: LSTM隐藏层的大小 + :param int num_classes: 一共有多少类 + """ + + def __init__(self, init_embed, hidden_size, num_classes): + super(SeqLabeling, self).__init__() + + self.Embedding = encoder.embedding.Embedding(init_embed) + self.Rnn = encoder.lstm.LSTM(self.Embedding.embedding_dim, hidden_size) + self.Linear = nn.Linear(hidden_size, num_classes) + self.Crf = decoder.crf.ConditionalRandomField(num_classes) + self.mask = None + + def forward(self, words, seq_len, target): + """ + :param torch.LongTensor words: [batch_size, max_len]，序列的index + :param torch.LongTensor seq_len: [batch_size,], 这个序列的长度 + :param torch.LongTensor target: [batch_size, max_len], 序列的目标值 + :return y: If truth is None, return list of [decode path(list)]. Used in testing and predicting. + If truth is not None, return loss, a scalar. Used in training. + """ + assert words.shape[0] == seq_len.shape[0] + assert target.shape == words.shape + self.mask = self._make_mask(words, seq_len) + + x = self.Embedding(words) + # [batch_size, max_len, word_emb_dim] + x, _ = self.Rnn(x, seq_len) + # [batch_size, max_len, hidden_size * direction] + x = self.Linear(x) + # [batch_size, max_len, num_classes] + return {C.LOSS: self._internal_loss(x, target)} + + def predict(self, words, seq_len): + """ + 用于在预测时使用 + + :param torch.LongTensor words: [batch_size, max_len] + :param torch.LongTensor seq_len: [batch_size,] + :return: {'pred': xx}, [batch_size, max_len] + """ + self.mask = self._make_mask(words, seq_len) + + x = self.Embedding(words) + # [batch_size, max_len, word_emb_dim] + x, _ = self.Rnn(x, seq_len) + # [batch_size, max_len, hidden_size * direction] + x = self.Linear(x) + # [batch_size, max_len, num_classes] + pred = self._decode(x) + return {C.OUTPUT: pred} + + def _internal_loss(self, x, y): + """ + Negative log likelihood loss. + :param x: Tensor, [batch_size, max_len, tag_size] + :param y: Tensor, [batch_size, max_len] + :return loss: a scalar Tensor + + """ + x = x.float() + y = y.long() + assert x.shape[:2] == y.shape + assert y.shape == self.mask.shape + total_loss = self.Crf(x, y, self.mask) + return torch.mean(total_loss) + + def _make_mask(self, x, seq_len): + batch_size, max_len = x.size(0), x.size(1) + mask = seq_len_to_mask(seq_len) + mask = mask.view(batch_size, max_len) + mask = mask.to(x).float() + return mask + + def _decode(self, x): + """ + :param torch.FloatTensor x: [batch_size, max_len, tag_size] + :return prediction: [batch_size, max_len] + """ + tag_seq, _ = self.Crf.viterbi_decode(x, self.mask) + return tag_seq + + +class AdvSeqLabel(nn.Module): + """ + 别名：:class:`fastNLP.models.AdvSeqLabel` :class:`fastNLP.models.sequence_labeling.AdvSeqLabel` + + 更复杂的Sequence Labelling模型。结构为Embedding, LayerNorm, 双向LSTM(两层)，FC，LayerNorm，DropOut，FC，CRF。 + + :param tuple(int,int),torch.FloatTensor,nn.Embedding,numpy.ndarray init_embed: Embedding的大小(传入tuple(int, int), + 第一个int为vocab_zie, 第二个int为embed_dim); 如果为Tensor, Embedding, ndarray等则直接使用该值初始化Embedding + :param int hidden_size: LSTM的隐层大小 + :param int num_classes: 有多少个类 + :param float dropout: LSTM中以及DropOut层的drop概率 + :param dict id2words: tag id转为其tag word的表。用于在CRF解码时防止解出非法的顺序，比如'BMES'这个标签规范中，'S' + 不能出现在'B'之后。这里也支持类似与'B-NN'，即'-'前为标签类型的指示，后面为具体的tag的情况。这里不但会保证 + 'B-NN'后面不为'S-NN'还会保证'B-NN'后面不会出现'M-xx'(任何非'M-NN'和'E-NN'的情况。) + :param str encoding_type: 支持"BIO", "BMES", "BEMSO", 只有在id2words不为None的情况有用。 + """ + + def __init__(self, init_embed, hidden_size, num_classes, dropout=0.3, id2words=None, encoding_type='bmes'): + + super().__init__() + + self.Embedding = encoder.embedding.Embedding(init_embed) + self.norm1 = torch.nn.LayerNorm(self.Embedding.embedding_dim) + self.Rnn = encoder.LSTM(input_size=self.Embedding.embedding_dim, hidden_size=hidden_size, num_layers=2, + dropout=dropout, + bidirectional=True, batch_first=True) + self.Linear1 = nn.Linear(hidden_size * 2, hidden_size * 2 // 3) + self.norm2 = torch.nn.LayerNorm(hidden_size * 2 // 3) + self.relu = torch.nn.LeakyReLU() + self.drop = torch.nn.Dropout(dropout) + self.Linear2 = nn.Linear(hidden_size * 2 // 3, num_classes) + + if id2words is None: + self.Crf = decoder.crf.ConditionalRandomField(num_classes, include_start_end_trans=False) + else: + self.Crf = decoder.crf.ConditionalRandomField(num_classes, include_start_end_trans=False, + allowed_transitions=allowed_transitions(id2words, + encoding_type=encoding_type)) + + def _decode(self, x): + """ + :param torch.FloatTensor x: [batch_size, max_len, tag_size] + :return torch.LongTensor, [batch_size, max_len] + """ + tag_seq, _ = self.Crf.viterbi_decode(x, self.mask) + return tag_seq + + def _internal_loss(self, x, y): + """ + Negative log likelihood loss. + :param x: Tensor, [batch_size, max_len, tag_size] + :param y: Tensor, [batch_size, max_len] + :return loss: a scalar Tensor + + """ + x = x.float() + y = y.long() + assert x.shape[:2] == y.shape + assert y.shape == self.mask.shape + total_loss = self.Crf(x, y, self.mask) + return torch.mean(total_loss) + + def _make_mask(self, x, seq_len): + batch_size, max_len = x.size(0), x.size(1) + mask = seq_len_to_mask(seq_len) + mask = mask.view(batch_size, max_len) + mask = mask.to(x).float() + return mask + + def _forward(self, words, seq_len, target=None): + """ + :param torch.LongTensor words: [batch_size, mex_len] + :param torch.LongTensor seq_len:[batch_size, ] + :param torch.LongTensor target: [batch_size, max_len] + :return y: If truth is None, return list of [decode path(list)]. Used in testing and predicting. + If truth is not None, return loss, a scalar. Used in training. + """ + + words = words.long() + seq_len = seq_len.long() + self.mask = self._make_mask(words, seq_len) + + # seq_len = seq_len.long() + target = target.long() if target is not None else None + + if next(self.parameters()).is_cuda: + words = words.cuda() + self.mask = self.mask.cuda() + + x = self.Embedding(words) + x = self.norm1(x) + # [batch_size, max_len, word_emb_dim] + + x, _ = self.Rnn(x, seq_len=seq_len) + + x = self.Linear1(x) + x = self.norm2(x) + x = self.relu(x) + x = self.drop(x) + x = self.Linear2(x) + if target is not None: + return {"loss": self._internal_loss(x, target)} + else: + return {"pred": self._decode(x)} + + def forward(self, words, seq_len, target): + """ + + :param torch.LongTensor words: [batch_size, mex_len] + :param torch.LongTensor seq_len: [batch_size, ] + :param torch.LongTensor target: [batch_size, max_len], 目标 + :return torch.Tensor: a scalar loss + """ + return self._forward(words, seq_len, target) + + def predict(self, words, seq_len): + """ + + :param torch.LongTensor words: [batch_size, mex_len] + :param torch.LongTensor seq_len: [batch_size, ] + :return torch.LongTensor: [batch_size, max_len] + """ + return self._forward(words, seq_len) diff --git a/fastNLP/models/sequence_modeling.py b/fastNLP/models/sequence_modeling.py deleted file mode 100644 index cb9e9478..00000000 --- a/fastNLP/models/sequence_modeling.py +++ /dev/null @@ -1,225 +0,0 @@ -import torch - -from fastNLP.models.base_model import BaseModel -from fastNLP.modules import decoder, encoder -from fastNLP.modules.decoder.CRF import allowed_transitions -from fastNLP.modules.utils import seq_mask - - -class SeqLabeling(BaseModel): - """ - PyTorch Network for sequence labeling - """ - - def __init__(self, args): - super(SeqLabeling, self).__init__() - vocab_size = args["vocab_size"] - word_emb_dim = args["word_emb_dim"] - hidden_dim = args["rnn_hidden_units"] - num_classes = args["num_classes"] - - self.Embedding = encoder.embedding.Embedding(vocab_size, word_emb_dim) - self.Rnn = encoder.lstm.LSTM(word_emb_dim, hidden_dim) - self.Linear = encoder.linear.Linear(hidden_dim, num_classes) - self.Crf = decoder.CRF.ConditionalRandomField(num_classes) - self.mask = None - - def forward(self, word_seq, word_seq_origin_len, truth=None): - """ - :param word_seq: LongTensor, [batch_size, mex_len] - :param word_seq_origin_len: LongTensor, [batch_size,], the origin lengths of the sequences. - :param truth: LongTensor, [batch_size, max_len] - :return y: If truth is None, return list of [decode path(list)]. Used in testing and predicting. - If truth is not None, return loss, a scalar. Used in training. - """ - assert word_seq.shape[0] == word_seq_origin_len.shape[0] - if truth is not None: - assert truth.shape == word_seq.shape - self.mask = self.make_mask(word_seq, word_seq_origin_len) - - x = self.Embedding(word_seq) - # [batch_size, max_len, word_emb_dim] - x = self.Rnn(x) - # [batch_size, max_len, hidden_size * direction] - x = self.Linear(x) - # [batch_size, max_len, num_classes] - return {"loss": self._internal_loss(x, truth) if truth is not None else None, - "predict": self.decode(x)} - - def loss(self, x, y): - """ Since the loss has been computed in forward(), this function simply returns x.""" - return x - - def _internal_loss(self, x, y): - """ - Negative log likelihood loss. - :param x: Tensor, [batch_size, max_len, tag_size] - :param y: Tensor, [batch_size, max_len] - :return loss: a scalar Tensor - - """ - x = x.float() - y = y.long() - assert x.shape[:2] == y.shape - assert y.shape == self.mask.shape - total_loss = self.Crf(x, y, self.mask) - return torch.mean(total_loss) - - def make_mask(self, x, seq_len): - batch_size, max_len = x.size(0), x.size(1) - mask = seq_mask(seq_len, max_len) - mask = mask.view(batch_size, max_len) - mask = mask.to(x).float() - return mask - - def decode(self, x, pad=True): - """ - :param x: FloatTensor, [batch_size, max_len, tag_size] - :param pad: pad the output sequence to equal lengths - :return prediction: list of [decode path(list)] - """ - max_len = x.shape[1] - tag_seq = self.Crf.viterbi_decode(x, self.mask) - # pad prediction to equal length - if pad is True: - for pred in tag_seq: - if len(pred) < max_len: - pred += [0] * (max_len - len(pred)) - return tag_seq - - -class AdvSeqLabel(SeqLabeling): - """ - Advanced Sequence Labeling Model - """ - - def __init__(self, args, emb=None, id2words=None): - super(AdvSeqLabel, self).__init__(args) - - vocab_size = args["vocab_size"] - word_emb_dim = args["word_emb_dim"] - hidden_dim = args["rnn_hidden_units"] - num_classes = args["num_classes"] - dropout = args['dropout'] - - self.Embedding = encoder.embedding.Embedding(vocab_size, word_emb_dim, init_emb=emb) - self.norm1 = torch.nn.LayerNorm(word_emb_dim) - # self.Rnn = encoder.lstm.LSTM(word_emb_dim, hidden_dim, num_layers=2, dropout=dropout, bidirectional=True) - self.Rnn = torch.nn.LSTM(input_size=word_emb_dim, hidden_size=hidden_dim, num_layers=2, dropout=dropout, - bidirectional=True, batch_first=True) - self.Linear1 = encoder.Linear(hidden_dim * 2, hidden_dim * 2 // 3) - self.norm2 = torch.nn.LayerNorm(hidden_dim * 2 // 3) - # self.batch_norm = torch.nn.BatchNorm1d(hidden_dim * 2 // 3) - self.relu = torch.nn.LeakyReLU() - self.drop = torch.nn.Dropout(dropout) - self.Linear2 = encoder.Linear(hidden_dim * 2 // 3, num_classes) - - if id2words is None: - self.Crf = decoder.CRF.ConditionalRandomField(num_classes, include_start_end_trans=False) - else: - self.Crf = decoder.CRF.ConditionalRandomField(num_classes, include_start_end_trans=False, - allowed_transitions=allowed_transitions(id2words, - encoding_type="bmes")) - - def forward(self, word_seq, word_seq_origin_len, truth=None): - """ - :param word_seq: LongTensor, [batch_size, mex_len] - :param word_seq_origin_len: LongTensor, [batch_size, ] - :param truth: LongTensor, [batch_size, max_len] - :return y: If truth is None, return list of [decode path(list)]. Used in testing and predicting. - If truth is not None, return loss, a scalar. Used in training. - """ - - word_seq = word_seq.long() - word_seq_origin_len = word_seq_origin_len.long() - self.mask = self.make_mask(word_seq, word_seq_origin_len) - sent_len, idx_sort = torch.sort(word_seq_origin_len, descending=True) - _, idx_unsort = torch.sort(idx_sort, descending=False) - - # word_seq_origin_len = word_seq_origin_len.long() - truth = truth.long() if truth is not None else None - - batch_size = word_seq.size(0) - max_len = word_seq.size(1) - if next(self.parameters()).is_cuda: - word_seq = word_seq.cuda() - idx_sort = idx_sort.cuda() - idx_unsort = idx_unsort.cuda() - self.mask = self.mask.cuda() - - x = self.Embedding(word_seq) - x = self.norm1(x) - # [batch_size, max_len, word_emb_dim] - - sent_variable = x[idx_sort] - sent_packed = torch.nn.utils.rnn.pack_padded_sequence(sent_variable, sent_len, batch_first=True) - - x, _ = self.Rnn(sent_packed) - # print(x) - # [batch_size, max_len, hidden_size * direction] - - sent_output = torch.nn.utils.rnn.pad_packed_sequence(x, batch_first=True)[0] - x = sent_output[idx_unsort] - - x = x.contiguous() - # x = x.view(batch_size * max_len, -1) - x = self.Linear1(x) - # x = self.batch_norm(x) - x = self.norm2(x) - x = self.relu(x) - x = self.drop(x) - x = self.Linear2(x) - # x = x.view(batch_size, max_len, -1) - # [batch_size, max_len, num_classes] - # TODO seq_lens的key这样做不合理 - return {"loss": self._internal_loss(x, truth) if truth is not None else None, - "predict": self.decode(x), - 'word_seq_origin_len': word_seq_origin_len} - - def predict(self, **x): - out = self.forward(**x) - return {"predict": out["predict"]} - - def loss(self, **kwargs): - assert 'loss' in kwargs - return kwargs['loss'] - - -if __name__ == '__main__': - args = { - 'vocab_size': 20, - 'word_emb_dim': 100, - 'rnn_hidden_units': 100, - 'num_classes': 10, - } - model = AdvSeqLabel(args) - data = [] - for i in range(20): - word_seq = torch.randint(20, (15,)).long() - word_seq_len = torch.LongTensor([15]) - truth = torch.randint(10, (15,)).long() - data.append((word_seq, word_seq_len, truth)) - optimizer = torch.optim.Adam(model.parameters(), lr=0.01) - print(model) - curidx = 0 - for i in range(1000): - endidx = min(len(data), curidx + 5) - b_word, b_len, b_truth = [], [], [] - for word_seq, word_seq_len, truth in data[curidx: endidx]: - b_word.append(word_seq) - b_len.append(word_seq_len) - b_truth.append(truth) - word_seq = torch.stack(b_word, dim=0) - word_seq_len = torch.cat(b_len, dim=0) - truth = torch.stack(b_truth, dim=0) - res = model(word_seq, word_seq_len, truth) - loss = res['loss'] - pred = res['predict'] - print('loss: {} acc {}'.format(loss.item(), - ((pred.data == truth).long().sum().float() / word_seq_len.sum().float()))) - optimizer.zero_grad() - loss.backward() - optimizer.step() - curidx = endidx - if curidx == len(data): - curidx = 0 diff --git a/fastNLP/models/snli.py b/fastNLP/models/snli.py index 6a7d8d84..395a9bbf 100644 --- a/fastNLP/models/snli.py +++ b/fastNLP/models/snli.py @@ -1,114 +1,152 @@ +__all__ = [ + "ESIM" +] + import torch import torch.nn as nn -import torch.nn.functional as F - -from fastNLP.models.base_model import BaseModel -from fastNLP.modules import decoder as Decoder -from fastNLP.modules import encoder as Encoder -from fastNLP.modules import aggregator as Aggregator +from .base_model import BaseModel +from ..core.const import Const +from ..modules import decoder as Decoder +from ..modules import encoder as Encoder +from ..modules import aggregator as Aggregator +from ..core.utils import seq_len_to_mask my_inf = 10e12 class ESIM(BaseModel): """ - PyTorch Network for SNLI task using ESIM model. - """ + 别名：:class:`fastNLP.models.ESIM` :class:`fastNLP.models.snli.ESIM` - def __init__(self, **kwargs): - super(ESIM, self).__init__() - self.vocab_size = kwargs["vocab_size"] - self.embed_dim = kwargs["embed_dim"] - self.hidden_size = kwargs["hidden_size"] - self.batch_first = kwargs["batch_first"] - self.dropout = kwargs["dropout"] - self.n_labels = kwargs["num_classes"] - self.gpu = kwargs["gpu"] and torch.cuda.is_available() + ESIM模型的一个PyTorch实现。 + ESIM模型的论文: Enhanced LSTM for Natural Language Inference (arXiv: 1609.06038) + :param int vocab_size: 词表大小 + :param int embed_dim: 词嵌入维度 + :param int hidden_size: LSTM隐层大小 + :param float dropout: dropout大小，默认为0 + :param int num_classes: 标签数目，默认为3 + :param numpy.array init_embedding: 初始词嵌入矩阵，形状为(vocab_size, embed_dim)，默认为None，即随机初始化词嵌入矩阵 + """ + + def __init__(self, vocab_size, embed_dim, hidden_size, dropout=0.0, num_classes=3, init_embedding=None): + + super(ESIM, self).__init__() + self.vocab_size = vocab_size + self.embed_dim = embed_dim + self.hidden_size = hidden_size + self.dropout = dropout + self.n_labels = num_classes + self.drop = nn.Dropout(self.dropout) - + self.embedding = Encoder.Embedding( - self.vocab_size, self.embed_dim, dropout=self.dropout, - init_emb=kwargs["init_embedding"] if "inin_embedding" in kwargs.keys() else None, + (self.vocab_size, self.embed_dim), dropout=self.dropout, ) - - self.embedding_layer = Encoder.Linear(self.embed_dim, self.hidden_size) - + + self.embedding_layer = nn.Linear(self.embed_dim, self.hidden_size) + self.encoder = Encoder.LSTM( input_size=self.embed_dim, hidden_size=self.hidden_size, num_layers=1, bias=True, - batch_first=self.batch_first, bidirectional=True + batch_first=True, bidirectional=True ) - - self.bi_attention = Aggregator.Bi_Attention() - self.mean_pooling = Aggregator.MeanPoolWithMask() + + self.bi_attention = Aggregator.BiAttention() + self.mean_pooling = Aggregator.AvgPoolWithMask() self.max_pooling = Aggregator.MaxPoolWithMask() - - self.inference_layer = Encoder.Linear(self.hidden_size * 4, self.hidden_size) - + + self.inference_layer = nn.Linear(self.hidden_size * 4, self.hidden_size) + self.decoder = Encoder.LSTM( input_size=self.hidden_size, hidden_size=self.hidden_size, num_layers=1, bias=True, - batch_first=self.batch_first, bidirectional=True + batch_first=True, bidirectional=True ) - + self.output = Decoder.MLP([4 * self.hidden_size, self.hidden_size, self.n_labels], 'tanh', dropout=self.dropout) - - def forward(self, premise, hypothesis, premise_len, hypothesis_len): + + def forward(self, words1, words2, seq_len1=None, seq_len2=None, target=None): """ Forward function - - :param premise: A Tensor represents premise: [batch size(B), premise seq len(PL)]. - :param hypothesis: A Tensor represents hypothesis: [B, hypothesis seq len(HL)]. - :param premise_len: A Tensor record which is a real word and which is a padding word in premise: [B, PL]. - :param hypothesis_len: A Tensor record which is a real word and which is a padding word in hypothesis: [B, HL]. - :return: prediction: A Dict with Tensor of classification result: [B, n_labels(N)]. + + :param torch.Tensor words1: [batch size(B), premise seq len(PL)] premise的token表示 + :param torch.Tensor words2: [B, hypothesis seq len(HL)] hypothesis的token表示 + :param torch.LongTensor seq_len1: [B] premise的长度 + :param torch.LongTensor seq_len2: [B] hypothesis的长度 + :param torch.LongTensor target: [B] 真实目标值 + :return: dict prediction: [B, n_labels(N)] 预测结果 """ - - premise0 = self.embedding_layer(self.embedding(premise)) - hypothesis0 = self.embedding_layer(self.embedding(hypothesis)) - + + premise0 = self.embedding_layer(self.embedding(words1)) + hypothesis0 = self.embedding_layer(self.embedding(words2)) + + if seq_len1 is not None: + seq_len1 = seq_len_to_mask(seq_len1) + else: + seq_len1 = torch.ones(premise0.size(0), premise0.size(1)) + seq_len1 = (seq_len1.long()).to(device=premise0.device) + if seq_len2 is not None: + seq_len2 = seq_len_to_mask(seq_len2) + else: + seq_len2 = torch.ones(hypothesis0.size(0), hypothesis0.size(1)) + seq_len2 = (seq_len2.long()).to(device=hypothesis0.device) + _BP, _PSL, _HP = premise0.size() _BH, _HSL, _HH = hypothesis0.size() - _BPL, _PLL = premise_len.size() - _HPL, _HLL = hypothesis_len.size() - + _BPL, _PLL = seq_len1.size() + _HPL, _HLL = seq_len2.size() + assert _BP == _BH and _BPL == _HPL and _BP == _BPL assert _HP == _HH assert _PSL == _PLL and _HSL == _HLL - + B, PL, H = premise0.size() B, HL, H = hypothesis0.size() - + a0 = self.encoder(self.drop(premise0)) # a0: [B, PL, H * 2] b0 = self.encoder(self.drop(hypothesis0)) # b0: [B, HL, H * 2] - + a = torch.mean(a0.view(B, PL, -1, H), dim=2) # a: [B, PL, H] b = torch.mean(b0.view(B, HL, -1, H), dim=2) # b: [B, HL, H] - - ai, bi = self.bi_attention(a, b, premise_len, hypothesis_len) - + + ai, bi = self.bi_attention(a, b, seq_len1, seq_len2) + ma = torch.cat((a, ai, a - ai, a * ai), dim=2) # ma: [B, PL, 4 * H] mb = torch.cat((b, bi, b - bi, b * bi), dim=2) # mb: [B, HL, 4 * H] - + f_ma = self.inference_layer(ma) f_mb = self.inference_layer(mb) - + vat = self.decoder(self.drop(f_ma)) vbt = self.decoder(self.drop(f_mb)) - + va = torch.mean(vat.view(B, PL, -1, H), dim=2) # va: [B, PL, H] vb = torch.mean(vbt.view(B, HL, -1, H), dim=2) # vb: [B, HL, H] - - va_ave = self.mean_pooling(va, premise_len, dim=1) # va_ave: [B, H] - va_max, va_arg_max = self.max_pooling(va, premise_len, dim=1) # va_max: [B, H] - vb_ave = self.mean_pooling(vb, hypothesis_len, dim=1) # vb_ave: [B, H] - vb_max, vb_arg_max = self.max_pooling(vb, hypothesis_len, dim=1) # vb_max: [B, H] - + + va_ave = self.mean_pooling(va, seq_len1, dim=1) # va_ave: [B, H] + va_max, va_arg_max = self.max_pooling(va, seq_len1, dim=1) # va_max: [B, H] + vb_ave = self.mean_pooling(vb, seq_len2, dim=1) # vb_ave: [B, H] + vb_max, vb_arg_max = self.max_pooling(vb, seq_len2, dim=1) # vb_max: [B, H] + v = torch.cat((va_ave, va_max, vb_ave, vb_max), dim=1) # v: [B, 4 * H] - - prediction = F.tanh(self.output(v)) # prediction: [B, N] - - return {'pred': prediction} - - def predict(self, premise, hypothesis, premise_len, hypothesis_len): - return self.forward(premise, hypothesis, premise_len, hypothesis_len) - + + prediction = torch.tanh(self.output(v)) # prediction: [B, N] + + if target is not None: + func = nn.CrossEntropyLoss() + loss = func(prediction, target) + return {Const.OUTPUT: prediction, Const.LOSS: loss} + + return {Const.OUTPUT: prediction} + + def predict(self, words1, words2, seq_len1=None, seq_len2=None, target=None): + """ Predict function + + :param torch.Tensor words1: [batch size(B), premise seq len(PL)] premise的token表示 + :param torch.Tensor words2: [B, hypothesis seq len(HL)] hypothesis的token表示 + :param torch.LongTensor seq_len1: [B] premise的长度 + :param torch.LongTensor seq_len2: [B] hypothesis的长度 + :param torch.LongTensor target: [B] 真实目标值 + :return: dict prediction: [B, n_labels(N)] 预测结果 + """ + prediction = self.forward(words1, words2, seq_len1, seq_len2)[Const.OUTPUT] + return {Const.OUTPUT: torch.argmax(prediction, dim=-1)} diff --git a/fastNLP/models/star_transformer.py b/fastNLP/models/star_transformer.py new file mode 100644 index 00000000..4c944a54 --- /dev/null +++ b/fastNLP/models/star_transformer.py @@ -0,0 +1,307 @@ +""" +Star-Transformer 的 Pytorch 实现。 +""" +__all__ = [ + "StarTransEnc", + "STNLICls", + "STSeqCls", + "STSeqLabel", +] + +import torch +from torch import nn + +from ..modules.encoder.star_transformer import StarTransformer +from ..core.utils import seq_len_to_mask +from ..modules.utils import get_embeddings +from ..core.const import Const + + +class StarTransEnc(nn.Module): + """ + 别名：:class:`fastNLP.models.StarTransEnc` :class:`fastNLP.models.star_transformer.StarTransEnc` + + 带word embedding的Star-Transformer Encoder + + :param init_embed: 单词词典, 可以是 tuple, 包括(num_embedings, embedding_dim), 即 + embedding的大小和每个词的维度. 也可以传入 nn.Embedding 对象, + 此时就以传入的对象作为embedding + :param hidden_size: 模型中特征维度. + :param num_layers: 模型层数. + :param num_head: 模型中multi-head的head个数. + :param head_dim: 模型中multi-head中每个head特征维度. + :param max_len: 模型能接受的最大输入长度. + :param emb_dropout: 词嵌入的dropout概率. + :param dropout: 模型除词嵌入外的dropout概率. + """ + + def __init__(self, init_embed, + hidden_size, + num_layers, + num_head, + head_dim, + max_len, + emb_dropout, + dropout): + super(StarTransEnc, self).__init__() + self.embedding = get_embeddings(init_embed) + emb_dim = self.embedding.embedding_dim + self.emb_fc = nn.Linear(emb_dim, hidden_size) + self.emb_drop = nn.Dropout(emb_dropout) + self.encoder = StarTransformer(hidden_size=hidden_size, + num_layers=num_layers, + num_head=num_head, + head_dim=head_dim, + dropout=dropout, + max_len=max_len) + + def forward(self, x, mask): + """ + :param FloatTensor x: [batch, length, hidden] 输入的序列 + :param ByteTensor mask: [batch, length] 输入序列的padding mask, 在没有内容(padding 部分) 为 0, + 否则为 1 + :return: [batch, length, hidden] 编码后的输出序列 + + [batch, hidden] 全局 relay 节点, 详见论文 + """ + x = self.embedding(x) + x = self.emb_fc(self.emb_drop(x)) + nodes, relay = self.encoder(x, mask) + return nodes, relay + + +class _Cls(nn.Module): + def __init__(self, in_dim, num_cls, hid_dim, dropout=0.1): + super(_Cls, self).__init__() + self.fc = nn.Sequential( + nn.Linear(in_dim, hid_dim), + nn.LeakyReLU(), + nn.Dropout(dropout), + nn.Linear(hid_dim, num_cls), + ) + + def forward(self, x): + h = self.fc(x) + return h + + +class _NLICls(nn.Module): + def __init__(self, in_dim, num_cls, hid_dim, dropout=0.1): + super(_NLICls, self).__init__() + self.fc = nn.Sequential( + nn.Dropout(dropout), + nn.Linear(in_dim * 4, hid_dim), # 4 + nn.LeakyReLU(), + nn.Dropout(dropout), + nn.Linear(hid_dim, num_cls), + ) + + def forward(self, x1, x2): + x = torch.cat([x1, x2, torch.abs(x1 - x2), x1 * x2], 1) + h = self.fc(x) + return h + + +class STSeqLabel(nn.Module): + """ + 别名：:class:`fastNLP.models.STSeqLabel` :class:`fastNLP.models.star_transformer.STSeqLabel` + + 用于序列标注的Star-Transformer模型 + + :param init_embed: 单词词典, 可以是 tuple, 包括(num_embedings, embedding_dim), 即 + embedding的大小和每个词的维度. 也可以传入 nn.Embedding 对象, + 此时就以传入的对象作为embedding + :param num_cls: 输出类别个数 + :param hidden_size: 模型中特征维度. Default: 300 + :param num_layers: 模型层数. Default: 4 + :param num_head: 模型中multi-head的head个数. Default: 8 + :param head_dim: 模型中multi-head中每个head特征维度. Default: 32 + :param max_len: 模型能接受的最大输入长度. Default: 512 + :param cls_hidden_size: 分类器隐层维度. Default: 600 + :param emb_dropout: 词嵌入的dropout概率. Default: 0.1 + :param dropout: 模型除词嵌入外的dropout概率. Default: 0.1 + """ + + def __init__(self, init_embed, num_cls, + hidden_size=300, + num_layers=4, + num_head=8, + head_dim=32, + max_len=512, + cls_hidden_size=600, + emb_dropout=0.1, + dropout=0.1, ): + super(STSeqLabel, self).__init__() + self.enc = StarTransEnc(init_embed=init_embed, + hidden_size=hidden_size, + num_layers=num_layers, + num_head=num_head, + head_dim=head_dim, + max_len=max_len, + emb_dropout=emb_dropout, + dropout=dropout) + self.cls = _Cls(hidden_size, num_cls, cls_hidden_size) + + def forward(self, words, seq_len): + """ + + :param words: [batch, seq_len] 输入序列 + :param seq_len: [batch,] 输入序列的长度 + :return output: [batch, num_cls, seq_len] 输出序列中每个元素的分类的概率 + """ + mask = seq_len_to_mask(seq_len) + nodes, _ = self.enc(words, mask) + output = self.cls(nodes) + output = output.transpose(1, 2) # make hidden to be dim 1 + return {Const.OUTPUT: output} # [bsz, n_cls, seq_len] + + def predict(self, words, seq_len): + """ + + :param words: [batch, seq_len] 输入序列 + :param seq_len: [batch,] 输入序列的长度 + :return output: [batch, seq_len] 输出序列中每个元素的分类 + """ + y = self.forward(words, seq_len) + _, pred = y[Const.OUTPUT].max(1) + return {Const.OUTPUT: pred} + + +class STSeqCls(nn.Module): + """ + 别名：:class:`fastNLP.models.STSeqCls` :class:`fastNLP.models.star_transformer.STSeqCls` + + 用于分类任务的Star-Transformer + + :param init_embed: 单词词典, 可以是 tuple, 包括(num_embedings, embedding_dim), 即 + embedding的大小和每个词的维度. 也可以传入 nn.Embedding 对象, + 此时就以传入的对象作为embedding + :param num_cls: 输出类别个数 + :param hidden_size: 模型中特征维度. Default: 300 + :param num_layers: 模型层数. Default: 4 + :param num_head: 模型中multi-head的head个数. Default: 8 + :param head_dim: 模型中multi-head中每个head特征维度. Default: 32 + :param max_len: 模型能接受的最大输入长度. Default: 512 + :param cls_hidden_size: 分类器隐层维度. Default: 600 + :param emb_dropout: 词嵌入的dropout概率. Default: 0.1 + :param dropout: 模型除词嵌入外的dropout概率. Default: 0.1 + """ + + def __init__(self, init_embed, num_cls, + hidden_size=300, + num_layers=4, + num_head=8, + head_dim=32, + max_len=512, + cls_hidden_size=600, + emb_dropout=0.1, + dropout=0.1, ): + super(STSeqCls, self).__init__() + self.enc = StarTransEnc(init_embed=init_embed, + hidden_size=hidden_size, + num_layers=num_layers, + num_head=num_head, + head_dim=head_dim, + max_len=max_len, + emb_dropout=emb_dropout, + dropout=dropout) + self.cls = _Cls(hidden_size, num_cls, cls_hidden_size) + + def forward(self, words, seq_len): + """ + + :param words: [batch, seq_len] 输入序列 + :param seq_len: [batch,] 输入序列的长度 + :return output: [batch, num_cls] 输出序列的分类的概率 + """ + mask = seq_len_to_mask(seq_len) + nodes, relay = self.enc(words, mask) + y = 0.5 * (relay + nodes.max(1)[0]) + output = self.cls(y) # [bsz, n_cls] + return {Const.OUTPUT: output} + + def predict(self, words, seq_len): + """ + + :param words: [batch, seq_len] 输入序列 + :param seq_len: [batch,] 输入序列的长度 + :return output: [batch, num_cls] 输出序列的分类 + """ + y = self.forward(words, seq_len) + _, pred = y[Const.OUTPUT].max(1) + return {Const.OUTPUT: pred} + + +class STNLICls(nn.Module): + """ + 别名：:class:`fastNLP.models.STNLICls` :class:`fastNLP.models.star_transformer.STNLICls` + + 用于自然语言推断(NLI)的Star-Transformer + + :param init_embed: 单词词典, 可以是 tuple, 包括(num_embedings, embedding_dim), 即 + embedding的大小和每个词的维度. 也可以传入 nn.Embedding 对象, + 此时就以传入的对象作为embedding + :param num_cls: 输出类别个数 + :param hidden_size: 模型中特征维度. Default: 300 + :param num_layers: 模型层数. Default: 4 + :param num_head: 模型中multi-head的head个数. Default: 8 + :param head_dim: 模型中multi-head中每个head特征维度. Default: 32 + :param max_len: 模型能接受的最大输入长度. Default: 512 + :param cls_hidden_size: 分类器隐层维度. Default: 600 + :param emb_dropout: 词嵌入的dropout概率. Default: 0.1 + :param dropout: 模型除词嵌入外的dropout概率. Default: 0.1 + """ + + def __init__(self, init_embed, num_cls, + hidden_size=300, + num_layers=4, + num_head=8, + head_dim=32, + max_len=512, + cls_hidden_size=600, + emb_dropout=0.1, + dropout=0.1, ): + super(STNLICls, self).__init__() + self.enc = StarTransEnc(init_embed=init_embed, + hidden_size=hidden_size, + num_layers=num_layers, + num_head=num_head, + head_dim=head_dim, + max_len=max_len, + emb_dropout=emb_dropout, + dropout=dropout) + self.cls = _NLICls(hidden_size, num_cls, cls_hidden_size) + + def forward(self, words1, words2, seq_len1, seq_len2): + """ + + :param words1: [batch, seq_len] 输入序列1 + :param words2: [batch, seq_len] 输入序列2 + :param seq_len1: [batch,] 输入序列1的长度 + :param seq_len2: [batch,] 输入序列2的长度 + :return output: [batch, num_cls] 输出分类的概率 + """ + mask1 = seq_len_to_mask(seq_len1) + mask2 = seq_len_to_mask(seq_len2) + + def enc(seq, mask): + nodes, relay = self.enc(seq, mask) + return 0.5 * (relay + nodes.max(1)[0]) + + y1 = enc(words1, mask1) + y2 = enc(words2, mask2) + output = self.cls(y1, y2) # [bsz, n_cls] + return {Const.OUTPUT: output} + + def predict(self, words1, words2, seq_len1, seq_len2): + """ + + :param words1: [batch, seq_len] 输入序列1 + :param words2: [batch, seq_len] 输入序列2 + :param seq_len1: [batch,] 输入序列1的长度 + :param seq_len2: [batch,] 输入序列2的长度 + :return output: [batch, num_cls] 输出分类的概率 + """ + y = self.forward(words1, words2, seq_len1, seq_len2) + _, pred = y[Const.OUTPUT].max(1) + return {Const.OUTPUT: pred} diff --git a/fastNLP/modules/__init__.py b/fastNLP/modules/__init__.py index 37223394..194fda4e 100644 --- a/fastNLP/modules/__init__.py +++ b/fastNLP/modules/__init__.py @@ -1,3 +1,51 @@ +""" +大部分用于的 NLP 任务神经网络都可以看做由编码 :mod:`~fastNLP.modules.encoder` 、 +聚合 :mod:`~fastNLP.modules.aggregator` 、解码 :mod:`~fastNLP.modules.decoder` 三种模块组成。 + +.. image:: figures/text_classification.png + +:mod:`~fastNLP.modules` 中实现了 fastNLP 提供的诸多模块组件，可以帮助用户快速搭建自己所需的网络。 +三种模块的功能和常见组件如下: + ++-----------------------+-----------------------+-----------------------+ +| module type | functionality | example | ++=======================+=======================+=======================+ +| encoder | 将输入编码为具有具 | embedding, RNN, CNN, | +| | 有表示能力的向量 | transformer | ++-----------------------+-----------------------+-----------------------+ +| aggregator | 从多个向量中聚合信息 | self-attention, | +| | | max-pooling | ++-----------------------+-----------------------+-----------------------+ +| decoder | 将具有某种表示意义的 | MLP, CRF | +| | 向量解码为需要的输出 | | +| | 形式 | | ++-----------------------+-----------------------+-----------------------+ + +""" +__all__ = [ + # "BertModel", + "ConvolutionCharEncoder", + "LSTMCharEncoder", + "ConvMaxpool", + "Embedding", + "LSTM", + "StarTransformer", + "TransformerEncoder", + "VarRNN", + "VarLSTM", + "VarGRU", + + "MaxPool", + "MaxPoolWithMask", + "AvgPool", + "MultiHeadAttention", + + "MLP", + "ConditionalRandomField", + "viterbi_decode", + "allowed_transitions", +] + from . import aggregator from . import decoder from . import encoder @@ -5,9 +53,4 @@ from .aggregator import * from .decoder import * from .dropout import TimestepDropout from .encoder import * - -__version__ = '0.0.0' - -__all__ = ['encoder', - 'decoder', - 'aggregator'] +from .utils import get_embeddings diff --git a/fastNLP/modules/aggregator/__init__.py b/fastNLP/modules/aggregator/__init__.py index 2fabb89e..a82138e7 100644 --- a/fastNLP/modules/aggregator/__init__.py +++ b/fastNLP/modules/aggregator/__init__.py @@ -1,10 +1,14 @@ -from .max_pool import MaxPool -from .max_pool import MaxPoolWithMask -from .avg_pool import AvgPool -from .avg_pool import MeanPoolWithMask -from .kmax_pool import KMaxPool +__all__ = [ + "MaxPool", + "MaxPoolWithMask", + "AvgPool", + + "MultiHeadAttention", +] -from .attention import Attention -from .attention import Bi_Attention -from .self_attention import SelfAttention +from .pooling import MaxPool +from .pooling import MaxPoolWithMask +from .pooling import AvgPool +from .pooling import AvgPoolWithMask +from .attention import MultiHeadAttention diff --git a/fastNLP/modules/aggregator/attention.py b/fastNLP/modules/aggregator/attention.py index 3fea1b10..4101b033 100644 --- a/fastNLP/modules/aggregator/attention.py +++ b/fastNLP/modules/aggregator/attention.py @@ -1,108 +1,233 @@ +__all__ = [ + "MultiHeadAttention" +] + import math import torch import torch.nn.functional as F from torch import nn -from fastNLP.modules.utils import mask_softmax - - -class Attention(torch.nn.Module): - def __init__(self, normalize=False): - super(Attention, self).__init__() - self.normalize = normalize - - def forward(self, query, memory, mask): - similarities = self._atten_forward(query, memory) - if self.normalize: - return mask_softmax(similarities, mask) - return similarities +from ..dropout import TimestepDropout - def _atten_forward(self, query, memory): - raise NotImplementedError +from ..utils import initial_parameter -class DotAtte(nn.Module): - def __init__(self, key_size, value_size): - super(DotAtte, self).__init__() +class DotAttention(nn.Module): + """ + .. todo:: + 补上文档 + """ + + def __init__(self, key_size, value_size, dropout=0): + super(DotAttention, self).__init__() self.key_size = key_size self.value_size = value_size self.scale = math.sqrt(key_size) - - def forward(self, Q, K, V, seq_mask=None): + self.drop = nn.Dropout(dropout) + self.softmax = nn.Softmax(dim=2) + + def forward(self, Q, K, V, mask_out=None): """ - :param Q: [batch, seq_len, key_size] - :param K: [batch, seq_len, key_size] - :param V: [batch, seq_len, value_size] - :param seq_mask: [batch, seq_len] + :param Q: [batch, seq_len_q, key_size] + :param K: [batch, seq_len_k, key_size] + :param V: [batch, seq_len_k, value_size] + :param mask_out: [batch, 1, seq_len] or [batch, seq_len_q, seq_len_k] """ output = torch.matmul(Q, K.transpose(1, 2)) / self.scale - if seq_mask is not None: - output.masked_fill_(seq_mask.lt(1), -float('inf')) - output = nn.functional.softmax(output, dim=2) + if mask_out is not None: + output.masked_fill_(mask_out, -1e8) + output = self.softmax(output) + output = self.drop(output) return torch.matmul(output, V) -class MultiHeadAtte(nn.Module): - def __init__(self, input_size, output_size, key_size, value_size, num_atte): - super(MultiHeadAtte, self).__init__() - self.in_linear = nn.ModuleList() - for i in range(num_atte * 3): - out_feat = key_size if (i % 3) != 2 else value_size - self.in_linear.append(nn.Linear(input_size, out_feat)) - self.attes = nn.ModuleList([DotAtte(key_size, value_size) for _ in range(num_atte)]) - self.out_linear = nn.Linear(value_size * num_atte, output_size) - - def forward(self, Q, K, V, seq_mask=None): - heads = [] - for i in range(len(self.attes)): - j = i * 3 - qi, ki, vi = self.in_linear[j](Q), self.in_linear[j+1](K), self.in_linear[j+2](V) - headi = self.attes[i](qi, ki, vi, seq_mask) - heads.append(headi) - output = torch.cat(heads, dim=2) - return self.out_linear(output) - - -class Bi_Attention(nn.Module): +class MultiHeadAttention(nn.Module): + """ + 别名：:class:`fastNLP.modules.MultiHeadAttention` :class:`fastNLP.modules.aggregator.attention.MultiHeadAttention` + + + :param input_size: int, 输入维度的大小。同时也是输出维度的大小。 + :param key_size: int, 每个head的维度大小。 + :param value_size: int，每个head中value的维度。 + :param num_head: int，head的数量。 + :param dropout: float。 + """ + + def __init__(self, input_size, key_size, value_size, num_head, dropout=0.1): + super(MultiHeadAttention, self).__init__() + self.input_size = input_size + self.key_size = key_size + self.value_size = value_size + self.num_head = num_head + + in_size = key_size * num_head + self.q_in = nn.Linear(input_size, in_size) + self.k_in = nn.Linear(input_size, in_size) + self.v_in = nn.Linear(input_size, in_size) + # follow the paper, do not apply dropout within dot-product + self.attention = DotAttention(key_size=key_size, value_size=value_size, dropout=0) + self.out = nn.Linear(value_size * num_head, input_size) + self.drop = TimestepDropout(dropout) + self.reset_parameters() + + def reset_parameters(self): + sqrt = math.sqrt + nn.init.normal_(self.q_in.weight, mean=0, std=sqrt(2.0 / (self.input_size + self.key_size))) + nn.init.normal_(self.k_in.weight, mean=0, std=sqrt(2.0 / (self.input_size + self.key_size))) + nn.init.normal_(self.v_in.weight, mean=0, std=sqrt(2.0 / (self.input_size + self.value_size))) + nn.init.xavier_normal_(self.out.weight) + + def forward(self, Q, K, V, atte_mask_out=None): + """ + + :param Q: [batch, seq_len_q, model_size] + :param K: [batch, seq_len_k, model_size] + :param V: [batch, seq_len_k, model_size] + :param seq_mask: [batch, seq_len] + """ + batch, sq, _ = Q.size() + sk = K.size(1) + d_k, d_v, n_head = self.key_size, self.value_size, self.num_head + # input linear + q = self.q_in(Q).view(batch, sq, n_head, d_k) + k = self.k_in(K).view(batch, sk, n_head, d_k) + v = self.v_in(V).view(batch, sk, n_head, d_v) + + # transpose q, k and v to do batch attention + q = q.permute(2, 0, 1, 3).contiguous().view(-1, sq, d_k) + k = k.permute(2, 0, 1, 3).contiguous().view(-1, sk, d_k) + v = v.permute(2, 0, 1, 3).contiguous().view(-1, sk, d_v) + if atte_mask_out is not None: + atte_mask_out = atte_mask_out.repeat(n_head, 1, 1) + atte = self.attention(q, k, v, atte_mask_out).view(n_head, batch, sq, d_v) + + # concat all heads, do output linear + atte = atte.permute(1, 2, 0, 3).contiguous().view(batch, sq, -1) + output = self.drop(self.out(atte)) + return output + + +class BiAttention(nn.Module): + r"""Bi Attention module + + .. todo:: + 这个模块的负责人来继续完善一下 + + Calculate Bi Attention matrix `e` + + .. math:: + + \begin{array}{ll} \\ + e_ij = {a}^{\mathbf{T}}_{i}{b}_{j} \\ + a_i = + b_j = + \end{array} + + """ + def __init__(self): - super(Bi_Attention, self).__init__() + super(BiAttention, self).__init__() self.inf = 10e12 - + def forward(self, in_x1, in_x2, x1_len, x2_len): - # in_x1: [batch_size, x1_seq_len, hidden_size] - # in_x2: [batch_size, x2_seq_len, hidden_size] - # x1_len: [batch_size, x1_seq_len] - # x2_len: [batch_size, x2_seq_len] - + """ + :param torch.Tensor in_x1: [batch_size, x1_seq_len, hidden_size] 第一句的特征表示 + :param torch.Tensor in_x2: [batch_size, x2_seq_len, hidden_size] 第二句的特征表示 + :param torch.Tensor x1_len: [batch_size, x1_seq_len] 第一句的0/1mask矩阵 + :param torch.Tensor x2_len: [batch_size, x2_seq_len] 第二句的0/1mask矩阵 + :return: torch.Tensor out_x1: [batch_size, x1_seq_len, hidden_size] 第一句attend到的特征表示 + torch.Tensor out_x2: [batch_size, x2_seq_len, hidden_size] 第一句attend到的特征表示 + + """ + assert in_x1.size()[0] == in_x2.size()[0] assert in_x1.size()[2] == in_x2.size()[2] # The batch size and hidden size must be equal. assert in_x1.size()[1] == x1_len.size()[1] and in_x2.size()[1] == x2_len.size()[1] # The seq len in in_x and x_len must be equal. assert in_x1.size()[0] == x1_len.size()[0] and x1_len.size()[0] == x2_len.size()[0] - + batch_size = in_x1.size()[0] x1_max_len = in_x1.size()[1] x2_max_len = in_x2.size()[1] - + in_x2_t = torch.transpose(in_x2, 1, 2) # [batch_size, hidden_size, x2_seq_len] - + attention_matrix = torch.bmm(in_x1, in_x2_t) # [batch_size, x1_seq_len, x2_seq_len] - + a_mask = x1_len.le(0.5).float() * -self.inf # [batch_size, x1_seq_len] a_mask = a_mask.view(batch_size, x1_max_len, -1) a_mask = a_mask.expand(-1, -1, x2_max_len) # [batch_size, x1_seq_len, x2_seq_len] b_mask = x2_len.le(0.5).float() * -self.inf b_mask = b_mask.view(batch_size, -1, x2_max_len) b_mask = b_mask.expand(-1, x1_max_len, -1) # [batch_size, x1_seq_len, x2_seq_len] - + attention_a = F.softmax(attention_matrix + a_mask, dim=2) # [batch_size, x1_seq_len, x2_seq_len] attention_b = F.softmax(attention_matrix + b_mask, dim=1) # [batch_size, x1_seq_len, x2_seq_len] - + out_x1 = torch.bmm(attention_a, in_x2) # [batch_size, x1_seq_len, hidden_size] attention_b_t = torch.transpose(attention_b, 1, 2) out_x2 = torch.bmm(attention_b_t, in_x1) # [batch_size, x2_seq_len, hidden_size] - + return out_x1, out_x2 + + +class SelfAttention(nn.Module): + """ + Self Attention Module. + + :param int input_size: 输入tensor的hidden维度 + :param int attention_unit: 输出tensor的hidden维度 + :param int attention_hops: + :param float drop: dropout概率，默认值为0.5 + :param str initial_method: 初始化参数方法 + """ + + def __init__(self, input_size, attention_unit=300, attention_hops=10, drop=0.5, initial_method=None, ): + super(SelfAttention, self).__init__() + + self.attention_hops = attention_hops + self.ws1 = nn.Linear(input_size, attention_unit, bias=False) + self.ws2 = nn.Linear(attention_unit, attention_hops, bias=False) + self.I = torch.eye(attention_hops, requires_grad=False) + self.I_origin = self.I + self.drop = nn.Dropout(drop) + self.tanh = nn.Tanh() + initial_parameter(self, initial_method) + + def _penalization(self, attention): + """ + compute the penalization term for attention module + """ + baz = attention.size(0) + size = self.I.size() + if len(size) != 3 or size[0] != baz: + self.I = self.I_origin.expand(baz, -1, -1) + self.I = self.I.to(device=attention.device) + attention_t = torch.transpose(attention, 1, 2).contiguous() + mat = torch.bmm(attention, attention_t) - self.I[:attention.size(0)] + ret = (torch.sum(torch.sum((mat ** 2), 2), 1).squeeze() + 1e-10) ** 0.5 + return torch.sum(ret) / size[0] + + def forward(self, input, input_origin): + """ + :param torch.Tensor input: [baz, senLen, h_dim] 要做attention的矩阵 + :param torch.Tensor input_origin: [baz , senLen] 原始token的index组成的矩阵，含有pad部分内容 + :return torch.Tensor output1: [baz, multi-head , h_dim] 经过attention操作后输入矩阵的结果 + :return torch.Tensor output2: [1] attention惩罚项，是一个标量 + """ + input = input.contiguous() + size = input.size() # [bsz, len, nhid] + + input_origin = input_origin.expand(self.attention_hops, -1, -1) # [hops,baz, len] + input_origin = input_origin.transpose(0, 1).contiguous() # [baz, hops,len] + + y1 = self.tanh(self.ws1(self.drop(input))) # [baz,len,dim] -->[bsz,len, attention-unit] + attention = self.ws2(y1).transpose(1, 2).contiguous() + # [bsz,len, attention-unit]--> [bsz, len, hop]--> [baz,hop,len] + + attention = attention + (-999999 * (input_origin == 0).float()) # remove the weight on padding token. + attention = F.softmax(attention, 2) # [baz ,hop, len] + return torch.bmm(attention, input), self._penalization(attention) # output1 --> [baz ,hop ,nhid] diff --git a/fastNLP/modules/aggregator/avg_pool.py b/fastNLP/modules/aggregator/avg_pool.py deleted file mode 100644 index e6f3fd4b..00000000 --- a/fastNLP/modules/aggregator/avg_pool.py +++ /dev/null @@ -1,36 +0,0 @@ -# python: 3.6 -# encoding: utf-8 - -import torch -import torch.nn as nn -import torch.nn.functional as F - - -class AvgPool(nn.Module): - """1-d average pooling module.""" - - def __init__(self, stride=None, padding=0): - super(AvgPool, self).__init__() - self.stride = stride - self.padding = padding - - def forward(self, x): - # [N,C,L] -> [N,C] - kernel_size = x.size(2) - x = F.max_pool1d( - input=x, - kernel_size=kernel_size, - stride=self.stride, - padding=self.padding) - return x.squeeze(dim=-1) - - -class MeanPoolWithMask(nn.Module): - def __init__(self): - super(MeanPoolWithMask, self).__init__() - self.inf = 10e12 - - def forward(self, tensor, mask, dim=0): - masks = mask.view(mask.size(0), mask.size(1), -1).float() - return torch.sum(tensor * masks, dim=dim) / torch.sum(masks, dim=1) - diff --git a/fastNLP/modules/aggregator/kmax_pool.py b/fastNLP/modules/aggregator/kmax_pool.py deleted file mode 100644 index 4d71130e..00000000 --- a/fastNLP/modules/aggregator/kmax_pool.py +++ /dev/null @@ -1,20 +0,0 @@ -# python: 3.6 -# encoding: utf-8 - -import torch -import torch.nn as nn -# import torch.nn.functional as F - - -class KMaxPool(nn.Module): - """K max-pooling module.""" - - def __init__(self, k=1): - super(KMaxPool, self).__init__() - self.k = k - - def forward(self, x): - # [N,C,L] -> [N,C*k] - x, index = torch.topk(x, self.k, dim=-1, sorted=False) - x = torch.reshape(x, (x.size(0), -1)) - return x diff --git a/fastNLP/modules/aggregator/max_pool.py b/fastNLP/modules/aggregator/max_pool.py deleted file mode 100644 index 60d68497..00000000 --- a/fastNLP/modules/aggregator/max_pool.py +++ /dev/null @@ -1,38 +0,0 @@ -# python: 3.6 -# encoding: utf-8 - -import torch -import torch.nn as nn -import torch.nn.functional as F - - -class MaxPool(nn.Module): - """1-d max-pooling module.""" - - def __init__(self, stride=None, padding=0, dilation=1): - super(MaxPool, self).__init__() - self.stride = stride - self.padding = padding - self.dilation = dilation - - def forward(self, x): - x = torch.transpose(x, 1, 2) # [N,L,C] -> [N,C,L] - kernel_size = x.size(2) - x = F.max_pool1d( # [N,L,C] -> [N,C,1] - input=x, - kernel_size=kernel_size, - stride=self.stride, - padding=self.padding, - dilation=self.dilation) - return x.squeeze(dim=-1) # [N,C,1] -> [N,C] - - -class MaxPoolWithMask(nn.Module): - def __init__(self): - super(MaxPoolWithMask, self).__init__() - self.inf = 10e12 - - def forward(self, tensor, mask, dim=0): - masks = mask.view(mask.size(0), mask.size(1), -1) - masks = masks.expand(-1, -1, tensor.size(2)).float() - return torch.max(tensor + masks.le(0.5).float() * -self.inf, dim=dim) diff --git a/fastNLP/modules/aggregator/pooling.py b/fastNLP/modules/aggregator/pooling.py new file mode 100644 index 00000000..51438aae --- /dev/null +++ b/fastNLP/modules/aggregator/pooling.py @@ -0,0 +1,146 @@ +__all__ = [ + "MaxPool", + "MaxPoolWithMask", + "AvgPool" +] +import torch +import torch.nn as nn + + +class MaxPool(nn.Module): + """ + 别名：:class:`fastNLP.modules.MaxPool` :class:`fastNLP.modules.aggregator.pooling.MaxPool` + + Max-pooling模块。 + + :param stride: 窗口移动大小，默认为kernel_size + :param padding: padding的内容，默认为0 + :param dilation: 控制窗口内元素移动距离的大小 + :param dimension: MaxPool的维度，支持1，2，3维。 + :param kernel_size: max pooling的窗口大小，默认为tensor最后k维，其中k为dimension + :param ceil_mode: + """ + + def __init__(self, stride=None, padding=0, dilation=1, dimension=1, kernel_size=None, ceil_mode=False): + + super(MaxPool, self).__init__() + assert (1 <= dimension) and (dimension <= 3) + self.dimension = dimension + self.stride = stride + self.padding = padding + self.dilation = dilation + self.kernel_size = kernel_size + self.ceil_mode = ceil_mode + + def forward(self, x): + if self.dimension == 1: + pooling = nn.MaxPool1d( + stride=self.stride, padding=self.padding, dilation=self.dilation, + kernel_size=self.kernel_size if self.kernel_size is not None else x.size(-1), + return_indices=False, ceil_mode=self.ceil_mode + ) + x = torch.transpose(x, 1, 2) # [N,L,C] -> [N,C,L] + elif self.dimension == 2: + pooling = nn.MaxPool2d( + stride=self.stride, padding=self.padding, dilation=self.dilation, + kernel_size=self.kernel_size if self.kernel_size is not None else (x.size(-2), x.size(-1)), + return_indices=False, ceil_mode=self.ceil_mode + ) + else: + pooling = nn.MaxPool2d( + stride=self.stride, padding=self.padding, dilation=self.dilation, + kernel_size=self.kernel_size if self.kernel_size is not None else (x.size(-3), x.size(-2), x.size(-1)), + return_indices=False, ceil_mode=self.ceil_mode + ) + x = pooling(x) + return x.squeeze(dim=-1) # [N,C,1] -> [N,C] + + +class MaxPoolWithMask(nn.Module): + """ + 别名：:class:`fastNLP.modules.MaxPoolWithMask` :class:`fastNLP.modules.aggregator.pooling.MaxPoolWithMask` + + 带mask矩阵的max pooling。在做max-pooling的时候不会考虑mask值为0的位置。 + """ + + def __init__(self): + super(MaxPoolWithMask, self).__init__() + self.inf = 10e12 + + def forward(self, tensor, mask, dim=1): + """ + :param torch.FloatTensor tensor: [batch_size, seq_len, channels] 初始tensor + :param torch.LongTensor mask: [batch_size, seq_len] 0/1的mask矩阵 + :param int dim: 需要进行max pooling的维度 + :return: + """ + masks = mask.view(mask.size(0), mask.size(1), -1) + masks = masks.expand(-1, -1, tensor.size(2)).float() + return torch.max(tensor + masks.le(0.5).float() * -self.inf, dim=dim)[0] + + +class KMaxPool(nn.Module): + """K max-pooling module.""" + + def __init__(self, k=1): + super(KMaxPool, self).__init__() + self.k = k + + def forward(self, x): + """ + :param torch.Tensor x: [N, C, L] 初始tensor + :return: torch.Tensor x: [N, C*k] k-max pool后的结果 + """ + x, index = torch.topk(x, self.k, dim=-1, sorted=False) + x = torch.reshape(x, (x.size(0), -1)) + return x + + +class AvgPool(nn.Module): + """ + 别名：:class:`fastNLP.modules.AvgPool` :class:`fastNLP.modules.aggregator.pooling.AvgPool` + + 给定形如[batch_size, max_len, hidden_size]的输入，在最后一维进行avg pooling. 输出为[batch_size, hidden_size] + """ + + def __init__(self, stride=None, padding=0): + super(AvgPool, self).__init__() + self.stride = stride + self.padding = padding + + def forward(self, x): + """ + :param torch.Tensor x: [N, C, L] 初始tensor + :return: torch.Tensor x: [N, C] avg pool后的结果 + """ + # [N,C,L] -> [N,C] + kernel_size = x.size(2) + pooling = nn.AvgPool1d( + kernel_size=kernel_size, + stride=self.stride, + padding=self.padding) + x = pooling(x) + return x.squeeze(dim=-1) + + +class AvgPoolWithMask(nn.Module): + """ + 别名：:class:`fastNLP.modules.AvgPoolWithMask` :class:`fastNLP.modules.aggregator.pooling.AvgPoolWithMask` + + 给定形如[batch_size, max_len, hidden_size]的输入，在最后一维进行avg pooling. 输出为[batch_size, hidden_size], pooling + 的时候只会考虑mask为1的位置 + """ + + def __init__(self): + super(AvgPoolWithMask, self).__init__() + self.inf = 10e12 + + def forward(self, tensor, mask, dim=1): + """ + :param torch.FloatTensor tensor: [batch_size, seq_len, channels] 初始tensor + :param torch.LongTensor mask: [batch_size, seq_len] 0/1的mask矩阵 + :param int dim: 需要进行max pooling的维度 + :return: + """ + masks = mask.view(mask.size(0), mask.size(1), -1).float() + return torch.sum(tensor * masks.float(), dim=dim) / torch.sum(masks.float(), dim=1) diff --git a/fastNLP/modules/aggregator/self_attention.py b/fastNLP/modules/aggregator/self_attention.py deleted file mode 100644 index b0f03791..00000000 --- a/fastNLP/modules/aggregator/self_attention.py +++ /dev/null @@ -1,68 +0,0 @@ -import torch -import torch.nn as nn -import torch.nn.functional as F -from torch.autograd import Variable - -from fastNLP.modules.utils import initial_parameter - - -class SelfAttention(nn.Module): - """Self Attention Module. - - :param int input_size: - :param int attention_unit: - :param int attention_hops: - :param float drop: - :param str initial_method: - :param bool use_cuda: - """ - - def __init__(self, input_size, attention_unit=350, attention_hops=10, drop=0.5, initial_method=None, - use_cuda=False): - super(SelfAttention, self).__init__() - - self.attention_hops = attention_hops - self.ws1 = nn.Linear(input_size, attention_unit, bias=False) - self.ws2 = nn.Linear(attention_unit, attention_hops, bias=False) - if use_cuda: - self.I = Variable(torch.eye(attention_hops).cuda(), requires_grad=False) - else: - self.I = Variable(torch.eye(attention_hops), requires_grad=False) - self.I_origin = self.I - self.drop = nn.Dropout(drop) - self.tanh = nn.Tanh() - initial_parameter(self, initial_method) - - def penalization(self, attention): - """ - compute the penalization term for attention module - """ - baz = attention.size(0) - size = self.I.size() - if len(size) != 3 or size[0] != baz: - self.I = self.I_origin.expand(baz, -1, -1) - attentionT = torch.transpose(attention, 1, 2).contiguous() - mat = torch.bmm(attention, attentionT) - self.I[:attention.size(0)] - ret = (torch.sum(torch.sum((mat ** 2), 2), 1).squeeze() + 1e-10) ** 0.5 - return torch.sum(ret) / size[0] - - def forward(self, input, input_origin): - """ - :param input: the matrix to do attention. [baz, senLen, h_dim] - :param inp: then token index include pad token( 0 ) [baz , senLen] - :return output1: the input matrix after attention operation [baz, multi-head , h_dim] - :return output2: the attention penalty term, a scalar [1] - """ - input = input.contiguous() - size = input.size() # [bsz, len, nhid] - - input_origin = input_origin.expand(self.attention_hops, -1, -1) # [hops,baz, len] - input_origin = input_origin.transpose(0, 1).contiguous() # [baz, hops,len] - - y1 = self.tanh(self.ws1(self.drop(input))) # [baz,len,dim] -->[bsz,len, attention-unit] - attention = self.ws2(y1).transpose(1, 2).contiguous() - # [bsz,len, attention-unit]--> [bsz, len, hop]--> [baz,hop,len] - - attention = attention + (-999999 * (input_origin == 0).float()) # remove the weight on padding token. - attention = F.softmax(attention, 2) # [baz ,hop, len] - return torch.bmm(attention, input), self.penalization(attention) # output1 --> [baz ,hop ,nhid] diff --git a/fastNLP/modules/decoder/MLP.py b/fastNLP/modules/decoder/MLP.py deleted file mode 100644 index c9198859..00000000 --- a/fastNLP/modules/decoder/MLP.py +++ /dev/null @@ -1,58 +0,0 @@ -import torch -import torch.nn as nn - -from fastNLP.modules.utils import initial_parameter - - -class MLP(nn.Module): - """Multilayer Perceptrons as a decoder - - :param list size_layer: list of int, define the size of MLP layers. - :param str activation: str or function, the activation function for hidden layers. - :param str initial_method: the name of initialization method. - :param float dropout: the probability of dropout. - - .. note:: - There is no activation function applying on output layer. - - """ - - def __init__(self, size_layer, activation='relu', initial_method=None, dropout=0.0): - super(MLP, self).__init__() - self.hiddens = nn.ModuleList() - self.output = None - for i in range(1, len(size_layer)): - if i + 1 == len(size_layer): - self.output = nn.Linear(size_layer[i-1], size_layer[i]) - else: - self.hiddens.append(nn.Linear(size_layer[i-1], size_layer[i])) - - self.dropout = nn.Dropout(p=dropout) - - actives = { - 'relu': nn.ReLU(), - 'tanh': nn.Tanh(), - } - if activation in actives: - self.hidden_active = actives[activation] - elif callable(activation): - self.hidden_active = activation - else: - raise ValueError("should set activation correctly: {}".format(activation)) - initial_parameter(self, initial_method) - - def forward(self, x): - for layer in self.hiddens: - x = self.dropout(self.hidden_active(layer(x))) - x = self.dropout(self.output(x)) - return x - - -if __name__ == '__main__': - net1 = MLP([5, 10, 5]) - net2 = MLP([5, 10, 5], 'tanh') - for net in [net1, net2]: - x = torch.randn(5, 5) - y = net(x) - print(x) - print(y) diff --git a/fastNLP/modules/decoder/__init__.py b/fastNLP/modules/decoder/__init__.py index a72b7cd0..664618b2 100644 --- a/fastNLP/modules/decoder/__init__.py +++ b/fastNLP/modules/decoder/__init__.py @@ -1,2 +1,11 @@ -from .CRF import ConditionalRandomField -from .MLP import MLP +__all__ = [ + "MLP", + "ConditionalRandomField", + "viterbi_decode", + "allowed_transitions" +] + +from .crf import ConditionalRandomField +from .mlp import MLP +from .utils import viterbi_decode +from .crf import allowed_transitions diff --git a/fastNLP/modules/decoder/CRF.py b/fastNLP/modules/decoder/crf.py similarity index 51% rename from fastNLP/modules/decoder/CRF.py rename to fastNLP/modules/decoder/crf.py index d7db3bf9..beb2b9be 100644 --- a/fastNLP/modules/decoder/CRF.py +++ b/fastNLP/modules/decoder/crf.py @@ -1,41 +1,37 @@ +__all__ = [ + "ConditionalRandomField", + "allowed_transitions" +] + import torch from torch import nn -from fastNLP.modules.utils import initial_parameter - - -def log_sum_exp(x, dim=-1): - max_value, _ = x.max(dim=dim, keepdim=True) - res = torch.log(torch.sum(torch.exp(x - max_value), dim=dim, keepdim=True)) + max_value - return res.squeeze(dim) - - -def seq_len_to_byte_mask(seq_lens): - # usually seq_lens: LongTensor, batch_size - # return value: ByteTensor, batch_size x max_len - batch_size = seq_lens.size(0) - max_len = seq_lens.max() - broadcast_arange = torch.arange(max_len).view(1, -1).repeat(batch_size, 1).to(seq_lens.device) - mask = broadcast_arange.float().lt(seq_lens.float().view(-1, 1)) - return mask +from ..utils import initial_parameter -def allowed_transitions(id2label, encoding_type='bio'): +def allowed_transitions(id2target, encoding_type='bio', include_start_end=True): """ + 别名：:class:`fastNLP.modules.allowed_transitions` :class:`fastNLP.modules.decoder.crf.allowed_transitions` - :param dict id2label: key是label的indices，value是str类型的tag或tag-label。value可以是只有tag的, 比如"B", "M"; 也可以是 - "B-NN", "M-NN", tag和label之间一定要用"-"隔开。一般可以通过Vocabulary.get_id2word()id2label。 - :param encoding_type: str, 支持"bio", "bmes"。 - :return: List[Tuple(int, int)]], 内部的Tuple是(from_tag_id, to_tag_id)。 返回的结果考虑了start和end，比如"BIO"中，B、O可以 - 位于序列的开端，而I不行。所以返回的结果中会包含(start_idx, B_idx), (start_idx, O_idx), 但是不包含(start_idx, I_idx). - start_idx=len(id2label), end_idx=len(id2label)+1。 + 给定一个id到label的映射表，返回所有可以跳转的(from_tag_id, to_tag_id)列表。 + + :param dict id2target: key是label的indices，value是str类型的tag或tag-label。value可以是只有tag的, 比如"B", "M"; 也可以是 + "B-NN", "M-NN", tag和label之间一定要用"-"隔开。一般可以通过Vocabulary.idx2word得到id2label。 + :param str encoding_type: 支持"bio", "bmes", "bmeso"。 + :param bool include_start_end: 是否包含开始与结尾的转换。比如在bio中，b/o可以在开头，但是i不能在开头； + 为True，返回的结果中会包含(start_idx, b_idx), (start_idx, o_idx), 但是不包含(start_idx, i_idx); + start_idx=len(id2label), end_idx=len(id2label)+1。为False, 返回的结果中不含与开始结尾相关的内容 + :return: List[Tuple(int, int)]], 内部的Tuple是可以进行跳转的(from_tag_id, to_tag_id)。 """ - num_tags = len(id2label) + num_tags = len(id2target) start_idx = num_tags end_idx = num_tags + 1 encoding_type = encoding_type.lower() allowed_trans = [] - id_label_lst = list(id2label.items()) + [(start_idx, 'start'), (end_idx, 'end')] + id_label_lst = list(id2target.items()) + if include_start_end: + id_label_lst += [(start_idx, 'start'), (end_idx, 'end')] + def split_tag_label(from_label): from_label = from_label.lower() if from_label in ['start', 'end']: @@ -45,7 +41,7 @@ def allowed_transitions(id2label, encoding_type='bio'): from_tag = from_label[:1] from_label = from_label[2:] return from_tag, from_label - + for from_id, from_label in id_label_lst: if from_label in ['', '']: continue @@ -54,22 +50,22 @@ def allowed_transitions(id2label, encoding_type='bio'): if to_label in ['', '']: continue to_tag, to_label = split_tag_label(to_label) - if is_transition_allowed(encoding_type, from_tag, from_label, to_tag, to_label): + if _is_transition_allowed(encoding_type, from_tag, from_label, to_tag, to_label): allowed_trans.append((from_id, to_id)) return allowed_trans -def is_transition_allowed(encoding_type, from_tag, from_label, to_tag, to_label): +def _is_transition_allowed(encoding_type, from_tag, from_label, to_tag, to_label): """ - :param encoding_type: str, 支持"BIO", "BMES"。 - :param from_tag: str, 比如"B", "M"之类的标注tag. 还包括start, end等两种特殊tag - :param from_label: str, 比如"PER", "LOC"等label - :param to_tag: str, 比如"B", "M"之类的标注tag. 还包括start, end等两种特殊tag - :param to_label: str, 比如"PER", "LOC"等label + :param str encoding_type: 支持"BIO", "BMES", "BEMSO"。 + :param str from_tag: 比如"B", "M"之类的标注tag. 还包括start, end等两种特殊tag + :param str from_label: 比如"PER", "LOC"等label + :param str to_tag: 比如"B", "M"之类的标注tag. 还包括start, end等两种特殊tag + :param str to_label: 比如"PER", "LOC"等label :return: bool，能否跃迁 """ - if to_tag=='start' or from_tag=='end': + if to_tag == 'start' or from_tag == 'end': return False encoding_type = encoding_type.lower() if encoding_type == 'bio': @@ -92,12 +88,12 @@ def is_transition_allowed(encoding_type, from_tag, from_label, to_tag, to_label) if from_tag == 'start': return to_tag in ('b', 'o') elif from_tag in ['b', 'i']: - return any([to_tag in ['end', 'b', 'o'], to_tag=='i' and from_label==to_label]) + return any([to_tag in ['end', 'b', 'o'], to_tag == 'i' and from_label == to_label]) elif from_tag == 'o': return to_tag in ['end', 'b', 'o'] else: raise ValueError("Unexpect tag {}. Expect only 'B', 'I', 'O'.".format(from_tag)) - + elif encoding_type == 'bmes': """ 第一行是to_tag, 第一列是from_tag，y任意条件下可转，-只有在label相同时可转，n不可转 @@ -120,55 +116,68 @@ def is_transition_allowed(encoding_type, from_tag, from_label, to_tag, to_label) if from_tag == 'start': return to_tag in ['b', 's'] elif from_tag == 'b': - return to_tag in ['m', 'e'] and from_label==to_label + return to_tag in ['m', 'e'] and from_label == to_label elif from_tag == 'm': - return to_tag in ['m', 'e'] and from_label==to_label + return to_tag in ['m', 'e'] and from_label == to_label elif from_tag in ['e', 's']: return to_tag in ['b', 's', 'end'] else: raise ValueError("Unexpect tag type {}. Expect only 'B', 'M', 'E', 'S'.".format(from_tag)) + elif encoding_type == 'bmeso': + if from_tag == 'start': + return to_tag in ['b', 's', 'o'] + elif from_tag == 'b': + return to_tag in ['m', 'e'] and from_label == to_label + elif from_tag == 'm': + return to_tag in ['m', 'e'] and from_label == to_label + elif from_tag in ['e', 's', 'o']: + return to_tag in ['b', 's', 'end', 'o'] + else: + raise ValueError("Unexpect tag type {}. Expect only 'B', 'M', 'E', 'S', 'O'.".format(from_tag)) + else: - raise ValueError("Only support BIO, BMES encoding type, got {}.".format(encoding_type)) + raise ValueError("Only support BIO, BMES, BMESO encoding type, got {}.".format(encoding_type)) class ConditionalRandomField(nn.Module): """ + 别名：:class:`fastNLP.modules.ConditionalRandomField` :class:`fastNLP.modules.decoder.crf.ConditionalRandomField` - :param int num_tags: 标签的数量。 - :param bool include_start_end_trans: 是否包含起始tag - :param list allowed_transitions: ``List[Tuple[from_tag_id(int), to_tag_id(int)]]``. 允许的跃迁，可以通过allowed_transitions()得到。 - 如果为None，则所有跃迁均为合法 - :param str initial_method: - """ + 条件随机场。 + 提供forward()以及viterbi_decode()两个方法，分别用于训练与inference。 - def __init__(self, num_tags, include_start_end_trans=False, allowed_transitions=None, initial_method=None): + :param int num_tags: 标签的数量 + :param bool include_start_end_trans: 是否考虑各个tag作为开始以及结尾的分数。 + :param List[Tuple[from_tag_id(int), to_tag_id(int)]] allowed_transitions: 内部的Tuple[from_tag_id(int), + to_tag_id(int)]视为允许发生的跃迁，其他没有包含的跃迁认为是禁止跃迁，可以通过 + allowed_transitions()函数得到；如果为None，则所有跃迁均为合法 + :param str initial_method: 初始化方法。见initial_parameter + """ + + def __init__(self, num_tags, include_start_end_trans=False, allowed_transitions=None, + initial_method=None): + super(ConditionalRandomField, self).__init__() - + self.include_start_end_trans = include_start_end_trans self.num_tags = num_tags - + # the meaning of entry in this matrix is (from_tag_id, to_tag_id) score self.trans_m = nn.Parameter(torch.randn(num_tags, num_tags)) if self.include_start_end_trans: self.start_scores = nn.Parameter(torch.randn(num_tags)) self.end_scores = nn.Parameter(torch.randn(num_tags)) - + if allowed_transitions is None: constrain = torch.zeros(num_tags + 2, num_tags + 2) else: - constrain = torch.ones(num_tags + 2, num_tags + 2) * -1000 + constrain = torch.full((num_tags + 2, num_tags + 2), fill_value=-10000.0, dtype=torch.float) for from_tag_id, to_tag_id in allowed_transitions: constrain[from_tag_id, to_tag_id] = 0 self._constrain = nn.Parameter(constrain, requires_grad=False) - - # self.reset_parameter() + initial_parameter(self, initial_method) - def reset_parameter(self): - nn.init.xavier_normal_(self.trans_m) - if self.include_start_end_trans: - nn.init.normal_(self.start_scores) - nn.init.normal_(self.end_scores) - + def _normalizer_likelihood(self, logits, mask): """Computes the (batch_size,) denominator term for the log-likelihood, which is the sum of the likelihoods across all possible state sequences. @@ -180,20 +189,23 @@ class ConditionalRandomField(nn.Module): seq_len, batch_size, n_tags = logits.size() alpha = logits[0] if self.include_start_end_trans: - alpha += self.start_scores.view(1, -1) - + alpha = alpha + self.start_scores.view(1, -1) + + flip_mask = mask.eq(0) + for i in range(1, seq_len): emit_score = logits[i].view(batch_size, 1, n_tags) trans_score = self.trans_m.view(1, n_tags, n_tags) tmp = alpha.view(batch_size, n_tags, 1) + emit_score + trans_score - alpha = log_sum_exp(tmp, 1) * mask[i].view(batch_size, 1) + alpha * (1 - mask[i]).view(batch_size, 1) - + alpha = torch.logsumexp(tmp, 1).masked_fill(flip_mask[i].view(batch_size, 1), 0) + \ + alpha.masked_fill(mask[i].byte().view(batch_size, 1), 0) + if self.include_start_end_trans: - alpha += self.end_scores.view(1, -1) - - return log_sum_exp(alpha, 1) - - def _glod_score(self, logits, tags, mask): + alpha = alpha + self.end_scores.view(1, -1) + + return torch.logsumexp(alpha, 1) + + def _gold_score(self, logits, tags, mask): """ Compute the score for the gold path. :param logits: FloatTensor, max_len x batch_size x num_tags @@ -204,98 +216,99 @@ class ConditionalRandomField(nn.Module): seq_len, batch_size, _ = logits.size() batch_idx = torch.arange(batch_size, dtype=torch.long, device=logits.device) seq_idx = torch.arange(seq_len, dtype=torch.long, device=logits.device) - + # trans_socre [L-1, B] - trans_score = self.trans_m[tags[:seq_len-1], tags[1:]] * mask[1:, :] + mask = mask.byte() + flip_mask = mask.eq(0) + trans_score = self.trans_m[tags[:seq_len - 1], tags[1:]].masked_fill(flip_mask[1:, :], 0) # emit_score [L, B] - emit_score = logits[seq_idx.view(-1,1), batch_idx.view(1,-1), tags] * mask + emit_score = logits[seq_idx.view(-1, 1), batch_idx.view(1, -1), tags].masked_fill(flip_mask, 0) # score [L-1, B] - score = trans_score + emit_score[:seq_len-1, :] - score = score.sum(0) + emit_score[-1] * mask[-1] + score = trans_score + emit_score[:seq_len - 1, :] + score = score.sum(0) + emit_score[-1].masked_fill(flip_mask[-1], 0) if self.include_start_end_trans: st_scores = self.start_scores.view(1, -1).repeat(batch_size, 1)[batch_idx, tags[0]] last_idx = mask.long().sum(0) - 1 ed_scores = self.end_scores.view(1, -1).repeat(batch_size, 1)[batch_idx, tags[last_idx, batch_idx]] - score += st_scores + ed_scores + score = score + st_scores + ed_scores # return [B,] return score - + def forward(self, feats, tags, mask): """ - Calculate the neg log likelihood - :param feats:FloatTensor, batch_size x max_len x num_tags - :param tags:LongTensor, batch_size x max_len - :param mask:ByteTensor batch_size x max_len - :return:FloatTensor, batch_size + 用于计算CRF的前向loss，返回值为一个batch_size的FloatTensor，可能需要mean()求得loss。 + + :param torch.FloatTensor feats: batch_size x max_len x num_tags，特征矩阵。 + :param torch.LongTensor tags: batch_size x max_len，标签矩阵。 + :param torch.ByteTensor mask: batch_size x max_len，为0的位置认为是padding。 + :return: torch.FloatTensor, (batch_size,) """ feats = feats.transpose(0, 1) tags = tags.transpose(0, 1).long() mask = mask.transpose(0, 1).float() all_path_score = self._normalizer_likelihood(feats, mask) - gold_path_score = self._glod_score(feats, tags, mask) - + gold_path_score = self._gold_score(feats, tags, mask) + return all_path_score - gold_path_score - - def viterbi_decode(self, data, mask, get_score=False, unpad=False): - """Given a feats matrix, return best decode path and best score. - - :param data:FloatTensor, batch_size x max_len x num_tags - :param mask:ByteTensor batch_size x max_len - :param get_score: bool, whether to output the decode score. - :param unpad: bool, 是否将结果unpad, - 如果False, 返回的是batch_size x max_len的tensor， - 如果True，返回的是List[List[int]], List[int]为每个sequence的label，已经unpadding了，即每个 - List[int]的长度是这个sample的有效长度 - :return: 如果get_score为False，返回结果根据unpadding变动 - 如果get_score为True, 返回 (paths, List[float], )。第一个仍然是解码后的路径(根据unpad变化)，第二个List[Float] - 为每个seqence的解码分数。 + + def viterbi_decode(self, logits, mask, unpad=False): + """给定一个特征矩阵以及转移分数矩阵，计算出最佳的路径以及对应的分数 + + :param torch.FloatTensor logits: batch_size x max_len x num_tags，特征矩阵。 + :param torch.ByteTensor mask: batch_size x max_len, 为0的位置认为是pad；如果为None，则认为没有padding。 + :param bool unpad: 是否将结果删去padding。False, 返回的是batch_size x max_len的tensor; True，返回的是 + List[List[int]], 内部的List[int]为每个sequence的label，已经除去pad部分，即每个List[int]的长度是这 + 个sample的有效长度。 + :return: 返回 (paths, scores)。 + paths: 是解码后的路径, 其值参照unpad参数. + scores: torch.FloatTensor, size为(batch_size,), 对应每个最优路径的分数。 """ - batch_size, seq_len, n_tags = data.size() - data = data.transpose(0, 1).data # L, B, H - mask = mask.transpose(0, 1).data.float() # L, B - + batch_size, seq_len, n_tags = logits.size() + logits = logits.transpose(0, 1).data # L, B, H + mask = mask.transpose(0, 1).data.byte() # L, B + # dp - vpath = data.new_zeros((seq_len, batch_size, n_tags), dtype=torch.long) - vscore = data[0] + vpath = logits.new_zeros((seq_len, batch_size, n_tags), dtype=torch.long) + vscore = logits[0] transitions = self._constrain.data.clone() transitions[:n_tags, :n_tags] += self.trans_m.data if self.include_start_end_trans: transitions[n_tags, :n_tags] += self.start_scores.data - transitions[:n_tags, n_tags+1] += self.end_scores.data - + transitions[:n_tags, n_tags + 1] += self.end_scores.data + vscore += transitions[n_tags, :n_tags] trans_score = transitions[:n_tags, :n_tags].view(1, n_tags, n_tags).data for i in range(1, seq_len): prev_score = vscore.view(batch_size, n_tags, 1) - cur_score = data[i].view(batch_size, 1, n_tags) + cur_score = logits[i].view(batch_size, 1, n_tags) score = prev_score + trans_score + cur_score best_score, best_dst = score.max(1) vpath[i] = best_dst - vscore = best_score * mask[i].view(batch_size, 1) + vscore * (1 - mask[i]).view(batch_size, 1) - - vscore += transitions[:n_tags, n_tags+1].view(1, -1) - + vscore = best_score.masked_fill(mask[i].eq(0).view(batch_size, 1), 0) + \ + vscore.masked_fill(mask[i].view(batch_size, 1), 0) + + if self.include_start_end_trans: + vscore += transitions[:n_tags, n_tags + 1].view(1, -1) + # backtrace - batch_idx = torch.arange(batch_size, dtype=torch.long, device=data.device) - seq_idx = torch.arange(seq_len, dtype=torch.long, device=data.device) + batch_idx = torch.arange(batch_size, dtype=torch.long, device=logits.device) + seq_idx = torch.arange(seq_len, dtype=torch.long, device=logits.device) lens = (mask.long().sum(0) - 1) # idxes [L, B], batched idx from seq_len-1 to 0 - idxes = (lens.view(1,-1) - seq_idx.view(-1,1)) % seq_len - - ans = data.new_empty((seq_len, batch_size), dtype=torch.long) + idxes = (lens.view(1, -1) - seq_idx.view(-1, 1)) % seq_len + + ans = logits.new_empty((seq_len, batch_size), dtype=torch.long) ans_score, last_tags = vscore.max(1) ans[idxes[0], batch_idx] = last_tags for i in range(seq_len - 1): last_tags = vpath[idxes[i], batch_idx, last_tags] - ans[idxes[i+1], batch_idx] = last_tags + ans[idxes[i + 1], batch_idx] = last_tags ans = ans.transpose(0, 1) if unpad: paths = [] for idx, seq_len in enumerate(lens): - paths.append(ans[idx, :seq_len+1].tolist()) + paths.append(ans[idx, :seq_len + 1].tolist()) else: paths = ans - if get_score: - return paths, ans_score.tolist() - return paths + return paths, ans_score diff --git a/fastNLP/modules/decoder/mlp.py b/fastNLP/modules/decoder/mlp.py new file mode 100644 index 00000000..c1579224 --- /dev/null +++ b/fastNLP/modules/decoder/mlp.py @@ -0,0 +1,96 @@ +__all__ = [ + "MLP" +] + +import torch +import torch.nn as nn + +from ..utils import initial_parameter + + +class MLP(nn.Module): + """ + 别名：:class:`fastNLP.modules.MLP` :class:`fastNLP.modules.decoder.mlp.MLP` + + 多层感知器 + + :param List[int] size_layer: 一个int的列表，用来定义MLP的层数，列表中的数字为每一层是hidden数目。MLP的层数为 len(size_layer) - 1 + :param Union[str,func,List[str]] activation: 一个字符串或者函数的列表，用来定义每一个隐层的激活函数，字符串包括relu，tanh和sigmoid，默认值为relu + :param Union[str,func] output_activation: 字符串或者函数，用来定义输出层的激活函数，默认值为None，表示输出层没有激活函数 + :param str initial_method: 参数初始化方式 + :param float dropout: dropout概率，默认值为0 + + .. note:: + 隐藏层的激活函数通过activation定义。一个str/function或者一个str/function的list可以被传入activation。 + 如果只传入了一个str/function，那么所有隐藏层的激活函数都由这个str/function定义； + 如果传入了一个str/function的list，那么每一个隐藏层的激活函数由这个list中对应的元素定义，其中list的长度为隐藏层数。 + 输出层的激活函数由output_activation定义，默认值为None，此时输出层没有激活函数。 + + Examples:: + + >>> net1 = MLP([5, 10, 5]) + >>> net2 = MLP([5, 10, 5], 'tanh') + >>> net3 = MLP([5, 6, 7, 8, 5], 'tanh') + >>> net4 = MLP([5, 6, 7, 8, 5], 'relu', output_activation='tanh') + >>> net5 = MLP([5, 6, 7, 8, 5], ['tanh', 'relu', 'tanh'], 'tanh') + >>> for net in [net1, net2, net3, net4, net5]: + >>> x = torch.randn(5, 5) + >>> y = net(x) + >>> print(x) + >>> print(y) + """ + + def __init__(self, size_layer, activation='relu', output_activation=None, initial_method=None, dropout=0.0): + super(MLP, self).__init__() + self.hiddens = nn.ModuleList() + self.output = None + self.output_activation = output_activation + for i in range(1, len(size_layer)): + if i + 1 == len(size_layer): + self.output = nn.Linear(size_layer[i - 1], size_layer[i]) + else: + self.hiddens.append(nn.Linear(size_layer[i - 1], size_layer[i])) + + self.dropout = nn.Dropout(p=dropout) + + actives = { + 'relu': nn.ReLU(), + 'tanh': nn.Tanh(), + 'sigmoid': nn.Sigmoid(), + } + if not isinstance(activation, list): + activation = [activation] * (len(size_layer) - 2) + elif len(activation) == len(size_layer) - 2: + pass + else: + raise ValueError( + f"the length of activation function list except {len(size_layer) - 2} but got {len(activation)}!") + self.hidden_active = [] + for func in activation: + if callable(activation): + self.hidden_active.append(activation) + elif func.lower() in actives: + self.hidden_active.append(actives[func]) + else: + raise ValueError("should set activation correctly: {}".format(activation)) + if self.output_activation is not None: + if callable(self.output_activation): + pass + elif self.output_activation.lower() in actives: + self.output_activation = actives[self.output_activation] + else: + raise ValueError("should set activation correctly: {}".format(activation)) + initial_parameter(self, initial_method) + + def forward(self, x): + """ + :param torch.Tensor x: MLP接受的输入 + :return: torch.Tensor : MLP的输出结果 + """ + for layer, func in zip(self.hiddens, self.hidden_active): + x = self.dropout(func(layer(x))) + x = self.output(x) + if self.output_activation is not None: + x = self.output_activation(x) + x = self.dropout(x) + return x diff --git a/fastNLP/modules/decoder/utils.py b/fastNLP/modules/decoder/utils.py new file mode 100644 index 00000000..249f3ff6 --- /dev/null +++ b/fastNLP/modules/decoder/utils.py @@ -0,0 +1,68 @@ +__all__ = [ + "viterbi_decode" +] +import torch + + +def viterbi_decode(logits, transitions, mask=None, unpad=False): + r""" + 别名：:class:`fastNLP.modules.viterbi_decode` :class:`fastNLP.modules.decoder.utils.viterbi_decode` + + 给定一个特征矩阵以及转移分数矩阵，计算出最佳的路径以及对应的分数 + + :param torch.FloatTensor logits: batch_size x max_len x num_tags，特征矩阵。 + :param torch.FloatTensor transitions: n_tags x n_tags。[i, j]位置的值认为是从tag i到tag j的转换。 + :param torch.ByteTensor mask: batch_size x max_len, 为0的位置认为是pad；如果为None，则认为没有padding。 + :param bool unpad: 是否将结果删去padding。False, 返回的是batch_size x max_len的tensor; True，返回的是 + List[List[int]], 内部的List[int]为每个sequence的label，已经除去pad部分，即每个List[int]的长度是这 + 个sample的有效长度。 + :return: 返回 (paths, scores)。 + paths: 是解码后的路径, 其值参照unpad参数. + scores: torch.FloatTensor, size为(batch_size,), 对应每个最优路径的分数。 + + """ + batch_size, seq_len, n_tags = logits.size() + assert n_tags == transitions.size(0) and n_tags == transitions.size( + 1), "The shapes of transitions and feats are not " \ + "compatible." + logits = logits.transpose(0, 1).data # L, B, H + if mask is not None: + mask = mask.transpose(0, 1).data.byte() # L, B + else: + mask = logits.new_ones((seq_len, batch_size), dtype=torch.uint8) + + # dp + vpath = logits.new_zeros((seq_len, batch_size, n_tags), dtype=torch.long) + vscore = logits[0] + + trans_score = transitions.view(1, n_tags, n_tags).data + for i in range(1, seq_len): + prev_score = vscore.view(batch_size, n_tags, 1) + cur_score = logits[i].view(batch_size, 1, n_tags) + score = prev_score + trans_score + cur_score + best_score, best_dst = score.max(1) + vpath[i] = best_dst + vscore = best_score.masked_fill(mask[i].eq(0).view(batch_size, 1), 0) + \ + vscore.masked_fill(mask[i].view(batch_size, 1), 0) + + # backtrace + batch_idx = torch.arange(batch_size, dtype=torch.long, device=logits.device) + seq_idx = torch.arange(seq_len, dtype=torch.long, device=logits.device) + lens = (mask.long().sum(0) - 1) + # idxes [L, B], batched idx from seq_len-1 to 0 + idxes = (lens.view(1, -1) - seq_idx.view(-1, 1)) % seq_len + + ans = logits.new_empty((seq_len, batch_size), dtype=torch.long) + ans_score, last_tags = vscore.max(1) + ans[idxes[0], batch_idx] = last_tags + for i in range(seq_len - 1): + last_tags = vpath[idxes[i], batch_idx, last_tags] + ans[idxes[i + 1], batch_idx] = last_tags + ans = ans.transpose(0, 1) + if unpad: + paths = [] + for idx, seq_len in enumerate(lens): + paths.append(ans[idx, :seq_len + 1].tolist()) + else: + paths = ans + return paths, ans_score diff --git a/fastNLP/modules/dropout.py b/fastNLP/modules/dropout.py index 34cf9e90..1363165c 100644 --- a/fastNLP/modules/dropout.py +++ b/fastNLP/modules/dropout.py @@ -1,11 +1,16 @@ +__all__ = [] + import torch class TimestepDropout(torch.nn.Dropout): - """This module accepts a ``[batch_size, num_timesteps, embedding_dim)]`` and use a single - dropout mask of shape ``(batch_size, embedding_dim)`` to apply on every time step. """ + 别名：:class:`fastNLP.modules.TimestepDropout` + 接受的参数shape为``[batch_size, num_timesteps, embedding_dim)]`` 使用同一个mask(shape为``(batch_size, embedding_dim)``) + 在每个timestamp上做dropout。 + """ + def forward(self, x): dropout_mask = x.new_ones(x.shape[0], x.shape[-1]) torch.nn.functional.dropout(dropout_mask, self.p, self.training, inplace=True) diff --git a/fastNLP/modules/encoder/__init__.py b/fastNLP/modules/encoder/__init__.py index b00a0ae9..bdc4cbf3 100644 --- a/fastNLP/modules/encoder/__init__.py +++ b/fastNLP/modules/encoder/__init__.py @@ -1,11 +1,28 @@ -from .conv import Conv +__all__ = [ + # "BertModel", + + "ConvolutionCharEncoder", + "LSTMCharEncoder", + + "ConvMaxpool", + + "Embedding", + + "LSTM", + + "StarTransformer", + + "TransformerEncoder", + + "VarRNN", + "VarLSTM", + "VarGRU" +] +from .bert import BertModel +from .char_encoder import ConvolutionCharEncoder, LSTMCharEncoder from .conv_maxpool import ConvMaxpool from .embedding import Embedding -from .linear import Linear from .lstm import LSTM - -__all__ = ["LSTM", - "Embedding", - "Linear", - "Conv", - "ConvMaxpool"] +from .star_transformer import StarTransformer +from .transformer import TransformerEncoder +from .variational_rnn import VarRNN, VarLSTM, VarGRU diff --git a/fastNLP/modules/encoder/bert.py b/fastNLP/modules/encoder/bert.py new file mode 100644 index 00000000..e123fda6 --- /dev/null +++ b/fastNLP/modules/encoder/bert.py @@ -0,0 +1,377 @@ +""" +bert.py is modified from huggingface/pytorch-pretrained-BERT, which is licensed under the Apache License 2.0. + +""" +import copy +import json +import math +import os + +import torch +from torch import nn + +CONFIG_FILE = 'bert_config.json' +MODEL_WEIGHTS = 'pytorch_model.bin' + + +def gelu(x): + return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0))) + + +def swish(x): + return x * torch.sigmoid(x) + + +ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish} + + +class BertLayerNorm(nn.Module): + def __init__(self, hidden_size, eps=1e-12): + super(BertLayerNorm, self).__init__() + self.weight = nn.Parameter(torch.ones(hidden_size)) + self.bias = nn.Parameter(torch.zeros(hidden_size)) + self.variance_epsilon = eps + + def forward(self, x): + u = x.mean(-1, keepdim=True) + s = (x - u).pow(2).mean(-1, keepdim=True) + x = (x - u) / torch.sqrt(s + self.variance_epsilon) + return self.weight * x + self.bias + + +class BertEmbeddings(nn.Module): + def __init__(self, vocab_size, hidden_size, max_position_embeddings, type_vocab_size, hidden_dropout_prob): + super(BertEmbeddings, self).__init__() + self.word_embeddings = nn.Embedding(vocab_size, hidden_size) + self.position_embeddings = nn.Embedding(max_position_embeddings, hidden_size) + self.token_type_embeddings = nn.Embedding(type_vocab_size, hidden_size) + + # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load + # any TensorFlow checkpoint file + self.LayerNorm = BertLayerNorm(hidden_size, eps=1e-12) + self.dropout = nn.Dropout(hidden_dropout_prob) + + def forward(self, input_ids, token_type_ids=None): + seq_length = input_ids.size(1) + position_ids = torch.arange(seq_length, dtype=torch.long, device=input_ids.device) + position_ids = position_ids.unsqueeze(0).expand_as(input_ids) + if token_type_ids is None: + token_type_ids = torch.zeros_like(input_ids) + + words_embeddings = self.word_embeddings(input_ids) + position_embeddings = self.position_embeddings(position_ids) + token_type_embeddings = self.token_type_embeddings(token_type_ids) + + embeddings = words_embeddings + position_embeddings + token_type_embeddings + embeddings = self.LayerNorm(embeddings) + embeddings = self.dropout(embeddings) + return embeddings + + +class BertSelfAttention(nn.Module): + def __init__(self, hidden_size, num_attention_heads, attention_probs_dropout_prob): + super(BertSelfAttention, self).__init__() + if hidden_size % num_attention_heads != 0: + raise ValueError( + "The hidden size (%d) is not a multiple of the number of attention " + "heads (%d)" % (hidden_size, num_attention_heads)) + self.num_attention_heads = num_attention_heads + self.attention_head_size = int(hidden_size / num_attention_heads) + self.all_head_size = self.num_attention_heads * self.attention_head_size + + self.query = nn.Linear(hidden_size, self.all_head_size) + self.key = nn.Linear(hidden_size, self.all_head_size) + self.value = nn.Linear(hidden_size, self.all_head_size) + + self.dropout = nn.Dropout(attention_probs_dropout_prob) + + def transpose_for_scores(self, x): + new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size) + x = x.view(*new_x_shape) + return x.permute(0, 2, 1, 3) + + def forward(self, hidden_states, attention_mask): + mixed_query_layer = self.query(hidden_states) + mixed_key_layer = self.key(hidden_states) + mixed_value_layer = self.value(hidden_states) + + query_layer = self.transpose_for_scores(mixed_query_layer) + key_layer = self.transpose_for_scores(mixed_key_layer) + value_layer = self.transpose_for_scores(mixed_value_layer) + + # Take the dot product between "query" and "key" to get the raw attention scores. + attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) + attention_scores = attention_scores / math.sqrt(self.attention_head_size) + # Apply the attention mask is (precomputed for all layers in BertModel forward() function) + attention_scores = attention_scores + attention_mask + + # Normalize the attention scores to probabilities. + attention_probs = nn.Softmax(dim=-1)(attention_scores) + + # This is actually dropping out entire tokens to attend to, which might + # seem a bit unusual, but is taken from the original Transformer paper. + attention_probs = self.dropout(attention_probs) + + context_layer = torch.matmul(attention_probs, value_layer) + context_layer = context_layer.permute(0, 2, 1, 3).contiguous() + new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,) + context_layer = context_layer.view(*new_context_layer_shape) + return context_layer + + +class BertSelfOutput(nn.Module): + def __init__(self, hidden_size, hidden_dropout_prob): + super(BertSelfOutput, self).__init__() + self.dense = nn.Linear(hidden_size, hidden_size) + self.LayerNorm = BertLayerNorm(hidden_size, eps=1e-12) + self.dropout = nn.Dropout(hidden_dropout_prob) + + def forward(self, hidden_states, input_tensor): + hidden_states = self.dense(hidden_states) + hidden_states = self.dropout(hidden_states) + hidden_states = self.LayerNorm(hidden_states + input_tensor) + return hidden_states + + +class BertAttention(nn.Module): + def __init__(self, hidden_size, num_attention_heads, attention_probs_dropout_prob, hidden_dropout_prob): + super(BertAttention, self).__init__() + self.self = BertSelfAttention(hidden_size, num_attention_heads, attention_probs_dropout_prob) + self.output = BertSelfOutput(hidden_size, hidden_dropout_prob) + + def forward(self, input_tensor, attention_mask): + self_output = self.self(input_tensor, attention_mask) + attention_output = self.output(self_output, input_tensor) + return attention_output + + +class BertIntermediate(nn.Module): + def __init__(self, hidden_size, intermediate_size, hidden_act): + super(BertIntermediate, self).__init__() + self.dense = nn.Linear(hidden_size, intermediate_size) + self.intermediate_act_fn = ACT2FN[hidden_act] \ + if isinstance(hidden_act, str) else hidden_act + + def forward(self, hidden_states): + hidden_states = self.dense(hidden_states) + hidden_states = self.intermediate_act_fn(hidden_states) + return hidden_states + + +class BertOutput(nn.Module): + def __init__(self, hidden_size, intermediate_size, hidden_dropout_prob): + super(BertOutput, self).__init__() + self.dense = nn.Linear(intermediate_size, hidden_size) + self.LayerNorm = BertLayerNorm(hidden_size, eps=1e-12) + self.dropout = nn.Dropout(hidden_dropout_prob) + + def forward(self, hidden_states, input_tensor): + hidden_states = self.dense(hidden_states) + hidden_states = self.dropout(hidden_states) + hidden_states = self.LayerNorm(hidden_states + input_tensor) + return hidden_states + + +class BertLayer(nn.Module): + def __init__(self, hidden_size, num_attention_heads, attention_probs_dropout_prob, hidden_dropout_prob, + intermediate_size, hidden_act): + super(BertLayer, self).__init__() + self.attention = BertAttention(hidden_size, num_attention_heads, attention_probs_dropout_prob, + hidden_dropout_prob) + self.intermediate = BertIntermediate(hidden_size, intermediate_size, hidden_act) + self.output = BertOutput(hidden_size, intermediate_size, hidden_dropout_prob) + + def forward(self, hidden_states, attention_mask): + attention_output = self.attention(hidden_states, attention_mask) + intermediate_output = self.intermediate(attention_output) + layer_output = self.output(intermediate_output, attention_output) + return layer_output + + +class BertEncoder(nn.Module): + def __init__(self, num_hidden_layers, hidden_size, num_attention_heads, attention_probs_dropout_prob, + hidden_dropout_prob, + intermediate_size, hidden_act): + super(BertEncoder, self).__init__() + layer = BertLayer(hidden_size, num_attention_heads, attention_probs_dropout_prob, hidden_dropout_prob, + intermediate_size, hidden_act) + self.layer = nn.ModuleList([copy.deepcopy(layer) for _ in range(num_hidden_layers)]) + + def forward(self, hidden_states, attention_mask, output_all_encoded_layers=True): + all_encoder_layers = [] + for layer_module in self.layer: + hidden_states = layer_module(hidden_states, attention_mask) + if output_all_encoded_layers: + all_encoder_layers.append(hidden_states) + if not output_all_encoded_layers: + all_encoder_layers.append(hidden_states) + return all_encoder_layers + + +class BertPooler(nn.Module): + def __init__(self, hidden_size): + super(BertPooler, self).__init__() + self.dense = nn.Linear(hidden_size, hidden_size) + self.activation = nn.Tanh() + + def forward(self, hidden_states): + # We "pool" the model by simply taking the hidden state corresponding + # to the first token. + first_token_tensor = hidden_states[:, 0] + pooled_output = self.dense(first_token_tensor) + pooled_output = self.activation(pooled_output) + return pooled_output + + +class BertModel(nn.Module): + """BERT(Bidirectional Embedding Representations from Transformers). + + 如果你想使用预训练好的权重矩阵，请在以下网址下载. + sources:: + + 'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz", + 'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gz", + 'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased.tar.gz", + 'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased.tar.gz", + 'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased.tar.gz", + 'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased.tar.gz", + 'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz", + + + 用预训练权重矩阵来建立BERT模型:: + + model = BertModel.from_pretrained("path/to/weights/directory") + + 用随机初始化权重矩阵来建立BERT模型:: + + model = BertModel() + + :param int vocab_size: 词表大小，默认值为30522，为BERT English uncase版本的词表大小 + :param int hidden_size: 隐层大小，默认值为768，为BERT base的版本 + :param int num_hidden_layers: 隐藏层数，默认值为12，为BERT base的版本 + :param int num_attention_heads: 多头注意力头数，默认值为12，为BERT base的版本 + :param int intermediate_size: FFN隐藏层大小，默认值是3072，为BERT base的版本 + :param str hidden_act: FFN隐藏层激活函数，默认值为``gelu`` + :param float hidden_dropout_prob: FFN隐藏层dropout，默认值为0.1 + :param float attention_probs_dropout_prob: Attention层的dropout，默认值为0.1 + :param int max_position_embeddings: 最大的序列长度，默认值为512， + :param int type_vocab_size: 最大segment数量，默认值为2 + :param int initializer_range: 初始化权重范围，默认值为0.02 + """ + + def __init__(self, vocab_size=30522, + hidden_size=768, + num_hidden_layers=12, + num_attention_heads=12, + intermediate_size=3072, + hidden_act="gelu", + hidden_dropout_prob=0.1, + attention_probs_dropout_prob=0.1, + max_position_embeddings=512, + type_vocab_size=2, + initializer_range=0.02, **kwargs): + super(BertModel, self).__init__() + self.embeddings = BertEmbeddings(vocab_size, hidden_size, max_position_embeddings, + type_vocab_size, hidden_dropout_prob) + self.encoder = BertEncoder(num_hidden_layers, hidden_size, num_attention_heads, + attention_probs_dropout_prob, hidden_dropout_prob, intermediate_size, + hidden_act) + self.pooler = BertPooler(hidden_size) + self.initializer_range = initializer_range + + self.apply(self.init_bert_weights) + + def init_bert_weights(self, module): + if isinstance(module, (nn.Linear, nn.Embedding)): + # Slightly different from the TF version which uses truncated_normal for initialization + # cf https://github.com/pytorch/pytorch/pull/5617 + module.weight.data.normal_(mean=0.0, std=self.initializer_range) + elif isinstance(module, BertLayerNorm): + module.bias.data.zero_() + module.weight.data.fill_(1.0) + if isinstance(module, nn.Linear) and module.bias is not None: + module.bias.data.zero_() + + def forward(self, input_ids, token_type_ids=None, attention_mask=None, output_all_encoded_layers=True): + if attention_mask is None: + attention_mask = torch.ones_like(input_ids) + if token_type_ids is None: + token_type_ids = torch.zeros_like(input_ids) + + # We create a 3D attention mask from a 2D tensor mask. + # Sizes are [batch_size, 1, 1, to_seq_length] + # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length] + # this attention mask is more simple than the triangular masking of causal attention + # used in OpenAI GPT, we just need to prepare the broadcast dimension here. + extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2) + + # Since attention_mask is 1.0 for positions we want to attend and 0.0 for + # masked positions, this operation will create a tensor which is 0.0 for + # positions we want to attend and -10000.0 for masked positions. + # Since we are adding it to the raw scores before the softmax, this is + # effectively the same as removing these entirely. + extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility + extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0 + + embedding_output = self.embeddings(input_ids, token_type_ids) + encoded_layers = self.encoder(embedding_output, + extended_attention_mask, + output_all_encoded_layers=output_all_encoded_layers) + sequence_output = encoded_layers[-1] + pooled_output = self.pooler(sequence_output) + if not output_all_encoded_layers: + encoded_layers = encoded_layers[-1] + return encoded_layers, pooled_output + + @classmethod + def from_pretrained(cls, pretrained_model_dir, state_dict=None, *inputs, **kwargs): + # Load config + config_file = os.path.join(pretrained_model_dir, CONFIG_FILE) + config = json.load(open(config_file, "r")) + # config = BertConfig.from_json_file(config_file) + # logger.info("Model config {}".format(config)) + # Instantiate model. + model = cls(*inputs, **config, **kwargs) + if state_dict is None: + weights_path = os.path.join(pretrained_model_dir, MODEL_WEIGHTS) + state_dict = torch.load(weights_path) + + old_keys = [] + new_keys = [] + for key in state_dict.keys(): + new_key = None + if 'gamma' in key: + new_key = key.replace('gamma', 'weight') + if 'beta' in key: + new_key = key.replace('beta', 'bias') + if new_key: + old_keys.append(key) + new_keys.append(new_key) + for old_key, new_key in zip(old_keys, new_keys): + state_dict[new_key] = state_dict.pop(old_key) + + missing_keys = [] + unexpected_keys = [] + error_msgs = [] + # copy state_dict so _load_from_state_dict can modify it + metadata = getattr(state_dict, '_metadata', None) + state_dict = state_dict.copy() + if metadata is not None: + state_dict._metadata = metadata + + def load(module, prefix=''): + local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {}) + module._load_from_state_dict( + state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs) + for name, child in module._modules.items(): + if child is not None: + load(child, prefix + name + '.') + + load(model, prefix='' if hasattr(model, 'bert') else 'bert.') + if len(missing_keys) > 0: + print("Weights of {} not initialized from pretrained model: {}".format( + model.__class__.__name__, missing_keys)) + if len(unexpected_keys) > 0: + print("Weights from pretrained model not used in {}: {}".format( + model.__class__.__name__, unexpected_keys)) + return model diff --git a/fastNLP/modules/encoder/char_embedding.py b/fastNLP/modules/encoder/char_embedding.py deleted file mode 100644 index 057d080c..00000000 --- a/fastNLP/modules/encoder/char_embedding.py +++ /dev/null @@ -1,82 +0,0 @@ -import torch -from torch import nn - -from fastNLP.modules.utils import initial_parameter - - -# from torch.nn.init import xavier_uniform -class ConvCharEmbedding(nn.Module): - """Character-level Embedding with CNN. - - :param int char_emb_size: the size of character level embedding. Default: 50 - say 26 characters, each embedded to 50 dim vector, then the input_size is 50. - :param tuple feature_maps: tuple of int. The length of the tuple is the number of convolution operations - over characters. The i-th integer is the number of filters (dim of out channels) for the i-th - convolution. - :param tuple kernels: tuple of int. The width of each kernel. - """ - - def __init__(self, char_emb_size=50, feature_maps=(40, 30, 30), kernels=(3, 4, 5), initial_method=None): - super(ConvCharEmbedding, self).__init__() - self.convs = nn.ModuleList([ - nn.Conv2d(1, feature_maps[i], kernel_size=(char_emb_size, kernels[i]), bias=True, padding=(0, 4)) - for i in range(len(kernels))]) - - initial_parameter(self, initial_method) - - def forward(self, x): - """ - :param x: ``[batch_size * sent_length, word_length, char_emb_size]`` - :return: feature map of shape [batch_size * sent_length, sum(feature_maps), 1] - """ - x = x.contiguous().view(x.size(0), 1, x.size(1), x.size(2)) - # [batch_size*sent_length, channel, width, height] - x = x.transpose(2, 3) - # [batch_size*sent_length, channel, height, width] - return self.convolute(x).unsqueeze(2) - - def convolute(self, x): - feats = [] - for conv in self.convs: - y = conv(x) - # [batch_size*sent_length, feature_maps[i], 1, width - kernels[i] + 1] - y = torch.squeeze(y, 2) - # [batch_size*sent_length, feature_maps[i], width - kernels[i] + 1] - y = torch.tanh(y) - y, __ = torch.max(y, 2) - # [batch_size*sent_length, feature_maps[i]] - feats.append(y) - return torch.cat(feats, 1) # [batch_size*sent_length, sum(feature_maps)] - - -class LSTMCharEmbedding(nn.Module): - """Character-level Embedding with LSTM. - - :param int char_emb_size: the size of character level embedding. Default: 50 - say 26 characters, each embedded to 50 dim vector, then the input_size is 50. - :param int hidden_size: the number of hidden units. Default: equal to char_emb_size. - """ - def __init__(self, char_emb_size=50, hidden_size=None, initial_method=None): - super(LSTMCharEmbedding, self).__init__() - self.hidden_size = char_emb_size if hidden_size is None else hidden_size - - self.lstm = nn.LSTM(input_size=char_emb_size, - hidden_size=self.hidden_size, - num_layers=1, - bias=True, - batch_first=True) - initial_parameter(self, initial_method) - - def forward(self, x): - """ - :param x: ``[ n_batch*n_word, word_length, char_emb_size]`` - :return: [ n_batch*n_word, char_emb_size] - """ - batch_size = x.shape[0] - h0 = torch.empty(1, batch_size, self.hidden_size) - h0 = nn.init.orthogonal_(h0) - c0 = torch.empty(1, batch_size, self.hidden_size) - c0 = nn.init.orthogonal_(c0) - - _, hidden = self.lstm(x, (h0, c0)) - return hidden[0].squeeze().unsqueeze(2) diff --git a/fastNLP/modules/encoder/char_encoder.py b/fastNLP/modules/encoder/char_encoder.py new file mode 100644 index 00000000..481ad7ad --- /dev/null +++ b/fastNLP/modules/encoder/char_encoder.py @@ -0,0 +1,96 @@ +__all__ = [ + "ConvolutionCharEncoder", + "LSTMCharEncoder" +] +import torch +import torch.nn as nn + +from ..utils import initial_parameter + + +# from torch.nn.init import xavier_uniform +class ConvolutionCharEncoder(nn.Module): + """ + 别名：:class:`fastNLP.modules.ConvolutionCharEncoder` :class:`fastNLP.modules.encoder.char_encoder.ConvolutionCharEncoder` + + char级别的卷积编码器. + + :param int char_emb_size: char级别embedding的维度. Default: 50 + :例: 有26个字符, 每一个的embedding是一个50维的向量, 所以输入的向量维度为50. + :param tuple feature_maps: 一个由int组成的tuple. tuple的长度是char级别卷积操作的数目, 第`i`个int表示第`i`个卷积操作的filter. + :param tuple kernels: 一个由int组成的tuple. tuple的长度是char级别卷积操作的数目, 第`i`个int表示第`i`个卷积操作的卷积核. + :param initial_method: 初始化参数的方式, 默认为`xavier normal` + """ + + def __init__(self, char_emb_size=50, feature_maps=(40, 30, 30), kernels=(3, 4, 5), initial_method=None): + super(ConvolutionCharEncoder, self).__init__() + self.convs = nn.ModuleList([ + nn.Conv2d(1, feature_maps[i], kernel_size=(char_emb_size, kernels[i]), bias=True, padding=(0, 4)) + for i in range(len(kernels))]) + + initial_parameter(self, initial_method) + + def forward(self, x): + """ + :param torch.Tensor x: ``[batch_size * sent_length, word_length, char_emb_size]`` 输入字符的embedding + :return: torch.Tensor : 卷积计算的结果, 维度为[batch_size * sent_length, sum(feature_maps), 1] + """ + x = x.contiguous().view(x.size(0), 1, x.size(1), x.size(2)) + # [batch_size*sent_length, channel, width, height] + x = x.transpose(2, 3) + # [batch_size*sent_length, channel, height, width] + return self._convolute(x).unsqueeze(2) + + def _convolute(self, x): + feats = [] + for conv in self.convs: + y = conv(x) + # [batch_size*sent_length, feature_maps[i], 1, width - kernels[i] + 1] + y = torch.squeeze(y, 2) + # [batch_size*sent_length, feature_maps[i], width - kernels[i] + 1] + y = torch.tanh(y) + y, __ = torch.max(y, 2) + # [batch_size*sent_length, feature_maps[i]] + feats.append(y) + return torch.cat(feats, 1) # [batch_size*sent_length, sum(feature_maps)] + + +class LSTMCharEncoder(nn.Module): + """ + 别名：:class:`fastNLP.modules.LSTMCharEncoder` :class:`fastNLP.modules.encoder.char_encoder.LSTMCharEncoder` + + char级别基于LSTM的encoder. + + + """ + + def __init__(self, char_emb_size=50, hidden_size=None, initial_method=None): + """ + :param int char_emb_size: char级别embedding的维度. Default: 50 + 例: 有26个字符, 每一个的embedding是一个50维的向量, 所以输入的向量维度为50. + :param int hidden_size: LSTM隐层的大小, 默认为char的embedding维度 + :param initial_method: 初始化参数的方式, 默认为`xavier normal` + """ + super(LSTMCharEncoder, self).__init__() + self.hidden_size = char_emb_size if hidden_size is None else hidden_size + + self.lstm = nn.LSTM(input_size=char_emb_size, + hidden_size=self.hidden_size, + num_layers=1, + bias=True, + batch_first=True) + initial_parameter(self, initial_method) + + def forward(self, x): + """ + :param torch.Tensor x: ``[ n_batch*n_word, word_length, char_emb_size]`` 输入字符的embedding + :return: torch.Tensor : [ n_batch*n_word, char_emb_size]经过LSTM编码的结果 + """ + batch_size = x.shape[0] + h0 = torch.empty(1, batch_size, self.hidden_size) + h0 = nn.init.orthogonal_(h0) + c0 = torch.empty(1, batch_size, self.hidden_size) + c0 = nn.init.orthogonal_(c0) + + _, hidden = self.lstm(x, (h0, c0)) + return hidden[0].squeeze().unsqueeze(2) diff --git a/fastNLP/modules/encoder/conv.py b/fastNLP/modules/encoder/conv.py deleted file mode 100644 index 42254a8b..00000000 --- a/fastNLP/modules/encoder/conv.py +++ /dev/null @@ -1,58 +0,0 @@ -# python: 3.6 -# encoding: utf-8 - -import torch -import torch.nn as nn - -from fastNLP.modules.utils import initial_parameter - - -# import torch.nn.functional as F - - -class Conv(nn.Module): - """Basic 1-d convolution module, initialized with xavier_uniform. - - :param int in_channels: - :param int out_channels: - :param tuple kernel_size: - :param int stride: - :param int padding: - :param int dilation: - :param int groups: - :param bool bias: - :param str activation: - :param str initial_method: - """ - def __init__(self, in_channels, out_channels, kernel_size, - stride=1, padding=0, dilation=1, - groups=1, bias=True, activation='relu', initial_method=None): - super(Conv, self).__init__() - self.conv = nn.Conv1d( - in_channels=in_channels, - out_channels=out_channels, - kernel_size=kernel_size, - stride=stride, - padding=padding, - dilation=dilation, - groups=groups, - bias=bias) - # xavier_uniform_(self.conv.weight) - - activations = { - 'relu': nn.ReLU(), - 'tanh': nn.Tanh()} - if activation in activations: - self.activation = activations[activation] - else: - raise Exception( - 'Should choose activation function from: ' + - ', '.join([x for x in activations])) - initial_parameter(self, initial_method) - - def forward(self, x): - x = torch.transpose(x, 1, 2) # [N,L,C] -> [N,C,L] - x = self.conv(x) # [N,C_in,L] -> [N,C_out,L] - x = self.activation(x) - x = torch.transpose(x, 1, 2) # [N,C,L] -> [N,L,C] - return x diff --git a/fastNLP/modules/encoder/conv_maxpool.py b/fastNLP/modules/encoder/conv_maxpool.py index 8b035871..ae6bea04 100644 --- a/fastNLP/modules/encoder/conv_maxpool.py +++ b/fastNLP/modules/encoder/conv_maxpool.py @@ -1,38 +1,50 @@ -# python: 3.6 -# encoding: utf-8 - +__all__ = [ + "ConvMaxpool" +] import torch import torch.nn as nn import torch.nn.functional as F -from fastNLP.modules.utils import initial_parameter +from ..utils import initial_parameter class ConvMaxpool(nn.Module): - """Convolution and max-pooling module with multiple kernel sizes. + """ + 别名：:class:`fastNLP.modules.ConvMaxpool` :class:`fastNLP.modules.encoder.conv_maxpool.ConvMaxpool` - :param int in_channels: - :param int out_channels: - :param tuple kernel_sizes: - :param int stride: - :param int padding: - :param int dilation: - :param int groups: - :param bool bias: - :param str activation: - :param str initial_method: + 集合了Convolution和Max-Pooling于一体的层。给定一个batch_size x max_len x input_size的输入，返回batch_size x + sum(output_channels) 大小的matrix。在内部，是先使用CNN给输入做卷积，然后经过activation激活层，在通过在长度(max_len) + 这一维进行max_pooling。最后得到每个sample的一个向量表示。 + + :param int in_channels: 输入channel的大小，一般是embedding的维度; 或encoder的output维度 + :param int,tuple(int) out_channels: 输出channel的数量。如果为list，则需要与kernel_sizes的数量保持一致 + :param int,tuple(int) kernel_sizes: 输出channel的kernel大小。 + :param int stride: 见pytorch Conv1D文档。所有kernel共享一个stride。 + :param int padding: 见pytorch Conv1D文档。所有kernel共享一个padding。 + :param int dilation: 见pytorch Conv1D文档。所有kernel共享一个dilation。 + :param int groups: 见pytorch Conv1D文档。所有kernel共享一个groups。 + :param bool bias: 见pytorch Conv1D文档。所有kernel共享一个bias。 + :param str activation: Convolution后的结果将通过该activation后再经过max-pooling。支持relu, sigmoid, tanh + :param str initial_method: str。 """ + def __init__(self, in_channels, out_channels, kernel_sizes, stride=1, padding=0, dilation=1, groups=1, bias=True, activation="relu", initial_method=None): super(ConvMaxpool, self).__init__() - + # convolution if isinstance(kernel_sizes, (list, tuple, int)): - if isinstance(kernel_sizes, int): + if isinstance(kernel_sizes, int) and isinstance(out_channels, int): out_channels = [out_channels] kernel_sizes = [kernel_sizes] - + elif isinstance(kernel_sizes, (tuple, list)) and isinstance(out_channels, (tuple, list)): + assert len(out_channels) == len( + kernel_sizes), "The number of out_channels should be equal to the number" \ + " of kernel_sizes." + else: + raise ValueError("The type of out_channels and kernel_sizes should be the same.") + self.convs = nn.ModuleList([nn.Conv1d( in_channels=in_channels, out_channels=oc, @@ -43,26 +55,39 @@ class ConvMaxpool(nn.Module): groups=groups, bias=bias) for oc, ks in zip(out_channels, kernel_sizes)]) - + else: raise Exception( 'Incorrect kernel sizes: should be list, tuple or int') - + # activation function if activation == 'relu': self.activation = F.relu + elif activation == 'sigmoid': + self.activation = F.sigmoid + elif activation == 'tanh': + self.activation = F.tanh else: raise Exception( - "Undefined activation function: choose from: relu") - + "Undefined activation function: choose from: relu, tanh, sigmoid") + initial_parameter(self, initial_method) + + def forward(self, x, mask=None): + """ - def forward(self, x): + :param torch.FloatTensor x: batch_size x max_len x input_size, 一般是经过embedding后的值 + :param mask: batch_size x max_len, pad的地方为0。不影响卷积运算，max-pool一定不会pool到pad为0的位置 + :return: + """ # [N,L,C] -> [N,C,L] x = torch.transpose(x, 1, 2) # convolution - xs = [self.activation(conv(x)) for conv in self.convs] # [[N,C,L]] + xs = [self.activation(conv(x)) for conv in self.convs] # [[N,C,L], ...] + if mask is not None: + mask = mask.unsqueeze(1) # B x 1 x L + xs = [x.masked_fill_(mask, float('-inf')) for x in xs] # max-pooling xs = [F.max_pool1d(input=i, kernel_size=i.size(2)).squeeze(2) - for i in xs] # [[N, C]] - return torch.cat(xs, dim=-1) # [N,C] + for i in xs] # [[N, C], ...] + return torch.cat(xs, dim=-1) # [N, C] diff --git a/fastNLP/modules/encoder/embedding.py b/fastNLP/modules/encoder/embedding.py index 7bcffb8e..c2dfab65 100644 --- a/fastNLP/modules/encoder/embedding.py +++ b/fastNLP/modules/encoder/embedding.py @@ -1,21 +1,50 @@ +__all__ = [ + "Embedding" +] import torch.nn as nn +from ..utils import get_embeddings -class Embedding(nn.Module): - """A simple lookup table. - - :param int nums: the size of the lookup table - :param int dims: the size of each vector - :param int padding_idx: pads the tensor with zeros whenever it encounters this index - :param bool sparse: If True, gradient matrix will be a sparse tensor. In this case, only optim.SGD(cuda and cpu) and optim.Adagrad(cpu) can be used +class Embedding(nn.Embedding): """ - def __init__(self, nums, dims, padding_idx=0, sparse=False, init_emb=None, dropout=0.0): - super(Embedding, self).__init__() - self.embed = nn.Embedding(nums, dims, padding_idx, sparse=sparse) - if init_emb is not None: - self.embed.weight = nn.Parameter(init_emb) - self.dropout = nn.Dropout(dropout) + 别名：:class:`fastNLP.modules.Embedding` :class:`fastNLP.modules.encoder.embedding.Embedding` + + Embedding组件. 可以通过self.num_embeddings获取词表大小; self.embedding_dim获取embedding的维度""" + + def __init__(self, init_embed, padding_idx=None, dropout=0.0, sparse=False, max_norm=None, norm_type=2, + scale_grad_by_freq=False): + """ + :param tuple(int,int),torch.FloatTensor,nn.Embedding,numpy.ndarray init_embed: Embedding的大小(传入tuple(int, int), + 第一个int为vocab_zie, 第二个int为embed_dim); 如果为Tensor, Embedding, ndarray等则直接使用该值初始化Embedding + :param None,int padding_idx: 该index的Embedding将一直为0. + :param float dropout: 对Embedding的输出的dropout。 + :param bool sparse: 如果为True，则对Embedding的梯度将是sparse的，参考Pytorch Embedding获取更多信息。 + :param None,float max_norm: 每个vector最大的norm能为多大 + :param int norm_type: norm的类型 + :param bool scale_grad_by_freq: 如果为True，将会把梯度除以这个词出现的次数. + """ + embed = get_embeddings(init_embed) + num_embeddings, embedding_dim = embed.weight.size() + + super().__init__(num_embeddings, embedding_dim, padding_idx=padding_idx, + max_norm=max_norm, norm_type=norm_type, scale_grad_by_freq=scale_grad_by_freq, + sparse=sparse, _weight=embed.weight.data) + del embed + + self.dropout = nn.Dropout(dropout) + def forward(self, x): - x = self.embed(x) + """ + :param torch.LongTensor x: [batch, seq_len] + :return: torch.Tensor : [batch, seq_len, embed_dim] + """ + x = super().forward(x) return self.dropout(x) + + def size(self): + """ + Embedding的大小 + :return: torch.Size() + """ + return self.weight.size() diff --git a/fastNLP/modules/encoder/linear.py b/fastNLP/modules/encoder/linear.py deleted file mode 100644 index 2dc31eea..00000000 --- a/fastNLP/modules/encoder/linear.py +++ /dev/null @@ -1,21 +0,0 @@ -import torch.nn as nn - -from fastNLP.modules.utils import initial_parameter - - -class Linear(nn.Module): - """ - - :param int input_size: input size - :param int output_size: output size - :param bool bias: - :param str initial_method: - """ - def __init__(self, input_size, output_size, bias=True, initial_method=None): - super(Linear, self).__init__() - self.linear = nn.Linear(input_size, output_size, bias) - initial_parameter(self, initial_method) - - def forward(self, x): - x = self.linear(x) - return x diff --git a/fastNLP/modules/encoder/lstm.py b/fastNLP/modules/encoder/lstm.py index 48c67a64..b4f960e7 100644 --- a/fastNLP/modules/encoder/lstm.py +++ b/fastNLP/modules/encoder/lstm.py @@ -1,39 +1,70 @@ +""" +轻量封装的 Pytorch LSTM 模块. +可在 forward 时传入序列的长度, 自动对padding做合适的处理. +""" +__all__ = [ + "LSTM" +] + +import torch import torch.nn as nn +import torch.nn.utils.rnn as rnn -from fastNLP.modules.utils import initial_parameter +from ..utils import initial_parameter class LSTM(nn.Module): - """Long Short Term Memory + """ + 别名：:class:`fastNLP.modules.LSTM` :class:`fastNLP.modules.encoder.lstm.LSTM` + + LSTM 模块, 轻量封装的Pytorch LSTM - :param int input_size: - :param int hidden_size: - :param int num_layers: - :param float dropout: - :param bool batch_first: - :param bool bidirectional: - :param bool bias: - :param str initial_method: - :param bool get_hidden: + :param input_size: 输入 `x` 的特征维度 + :param hidden_size: 隐状态 `h` 的特征维度 + :param num_layers: rnn的层数. Default: 1 + :param dropout: 层间dropout概率. Default: 0 + :param bidirectional: 若为 ``True``, 使用双向的RNN. Default: ``False`` + :param batch_first: 若为 ``True``, 输入和输出 ``Tensor`` 形状为 + :(batch, seq, feature). Default: ``False`` + :param bias: 如果为 ``False``, 模型将不会使用bias. Default: ``True`` """ + def __init__(self, input_size, hidden_size=100, num_layers=1, dropout=0.0, batch_first=True, - bidirectional=False, bias=True, initial_method=None, get_hidden=False): + bidirectional=False, bias=True, initial_method=None): super(LSTM, self).__init__() + self.batch_first = batch_first self.lstm = nn.LSTM(input_size, hidden_size, num_layers, bias=bias, batch_first=batch_first, dropout=dropout, bidirectional=bidirectional) - self.get_hidden = get_hidden initial_parameter(self, initial_method) + + def forward(self, x, seq_len=None, h0=None, c0=None): + """ - def forward(self, x, h0=None, c0=None): + :param x: [batch, seq_len, input_size] 输入序列 + :param seq_len: [batch, ] 序列长度, 若为 ``None``, 所有输入看做一样长. Default: ``None`` + :param h0: [batch, hidden_size] 初始隐状态, 若为 ``None`` , 设为全1向量. Default: ``None`` + :param c0: [batch, hidden_size] 初始Cell状态, 若为 ``None`` , 设为全1向量. Default: ``None`` + :return (output, ht) 或 output: 若 ``get_hidden=True`` [batch, seq_len, hidden_size*num_direction] 输出序列 + 和 [batch, hidden_size*num_direction] 最后时刻隐状态. + """ if h0 is not None and c0 is not None: - x, (ht, ct) = self.lstm(x, (h0, c0)) + hx = (h0, c0) else: - x, (ht, ct) = self.lstm(x) - if self.get_hidden: - return x, (ht, ct) + hx = None + if seq_len is not None and not isinstance(x, rnn.PackedSequence): + sort_lens, sort_idx = torch.sort(seq_len, dim=0, descending=True) + if self.batch_first: + x = x[sort_idx] + else: + x = x[:, sort_idx] + x = rnn.pack_padded_sequence(x, sort_lens, batch_first=self.batch_first) + output, hx = self.lstm(x, hx) # -> [N,L,C] + output, _ = rnn.pad_packed_sequence(output, batch_first=self.batch_first) + _, unsort_idx = torch.sort(sort_idx, dim=0, descending=False) + if self.batch_first: + output = output[unsort_idx] + else: + output = output[:, unsort_idx] else: - return x - - -if __name__ == "__main__": - lstm = LSTM(10) + output, hx = self.lstm(x, hx) + return output, hx diff --git a/fastNLP/modules/encoder/masked_rnn.py b/fastNLP/modules/encoder/masked_rnn.py deleted file mode 100644 index 321546c4..00000000 --- a/fastNLP/modules/encoder/masked_rnn.py +++ /dev/null @@ -1,424 +0,0 @@ -__author__ = 'max' - -import torch -import torch.nn as nn -import torch.nn.functional as F - -from fastNLP.modules.utils import initial_parameter - - -def MaskedRecurrent(reverse=False): - def forward(input, hidden, cell, mask, train=True, dropout=0): - """ - :param input: - :param hidden: - :param cell: - :param mask: - :param dropout: step之间的dropout，对mask了的也会drop，应该是没问题的，反正没有gradient - :param train: 控制dropout的行为，在StackedRNN的forward中调用 - :return: - """ - output = [] - steps = range(input.size(0) - 1, -1, -1) if reverse else range(input.size(0)) - for i in steps: - if mask is None or mask[i].data.min() > 0.5: # 没有mask，都是1 - hidden = cell(input[i], hidden) - elif mask[i].data.max() > 0.5: # 有mask，但不全为0 - hidden_next = cell(input[i], hidden) # 一次喂入一个batch！ - # hack to handle LSTM - if isinstance(hidden, tuple): # LSTM outputs a tuple of (hidden, cell), this is a common hack 😁 - mask = mask.float() - hx, cx = hidden - hp1, cp1 = hidden_next - hidden = ( - hx + (hp1 - hx) * mask[i].squeeze(), - cx + (cp1 - cx) * mask[i].squeeze()) # Why? 我知道了！！如果是mask就不用改变 - else: - hidden = hidden + (hidden_next - hidden) * mask[i] - - # if dropout != 0 and train: # warning, should i treat masked tensor differently? - # if isinstance(hidden, tuple): - # hidden = (F.dropout(hidden[0], p=dropout, training=train), - # F.dropout(hidden[1], p=dropout, training=train)) - # else: - # hidden = F.dropout(hidden, p=dropout, training=train) - - # hack to handle LSTM - output.append(hidden[0] if isinstance(hidden, tuple) else hidden) - - if reverse: - output.reverse() - output = torch.cat(output, 0).view(input.size(0), *output[0].size()) - - return hidden, output - - return forward - - -def StackedRNN(inners, num_layers, lstm=False, train=True, step_dropout=0, layer_dropout=0): - num_directions = len(inners) # rec_factory! - total_layers = num_layers * num_directions - - def forward(input, hidden, cells, mask): - assert (len(cells) == total_layers) - next_hidden = [] - - if lstm: - hidden = list(zip(*hidden)) - - for i in range(num_layers): - all_output = [] - for j, inner in enumerate(inners): - l = i * num_directions + j - hy, output = inner(input, hidden[l], cells[l], mask, step_dropout, train) - next_hidden.append(hy) - all_output.append(output) - - input = torch.cat(all_output, input.dim() - 1) # 下一层的输入 - - if layer_dropout != 0 and i < num_layers - 1: - input = F.dropout(input, p=layer_dropout, training=train, inplace=False) - - if lstm: - next_h, next_c = zip(*next_hidden) - next_hidden = ( - torch.cat(next_h, 0).view(total_layers, *next_h[0].size()), - torch.cat(next_c, 0).view(total_layers, *next_c[0].size()) - ) - else: - next_hidden = torch.cat(next_hidden, 0).view(total_layers, *next_hidden[0].size()) - - return next_hidden, input - - return forward - - -def AutogradMaskedRNN(num_layers=1, batch_first=False, train=True, layer_dropout=0, step_dropout=0, - bidirectional=False, lstm=False): - rec_factory = MaskedRecurrent - - if bidirectional: - layer = (rec_factory(), rec_factory(reverse=True)) - else: - layer = (rec_factory(),) # rec_factory 就是每层的结构啦！！在MaskedRecurrent中进行每层的计算！然后用StackedRNN接起来 - - func = StackedRNN(layer, - num_layers, - lstm=lstm, - layer_dropout=layer_dropout, step_dropout=step_dropout, - train=train) - - def forward(input, cells, hidden, mask): - if batch_first: - input = input.transpose(0, 1) - if mask is not None: - mask = mask.transpose(0, 1) - - nexth, output = func(input, hidden, cells, mask) - - if batch_first: - output = output.transpose(0, 1) - - return output, nexth - - return forward - - -def MaskedStep(): - def forward(input, hidden, cell, mask): - if mask is None or mask.data.min() > 0.5: - hidden = cell(input, hidden) - elif mask.data.max() > 0.5: - hidden_next = cell(input, hidden) - # hack to handle LSTM - if isinstance(hidden, tuple): - hx, cx = hidden - hp1, cp1 = hidden_next - hidden = (hx + (hp1 - hx) * mask, cx + (cp1 - cx) * mask) - else: - hidden = hidden + (hidden_next - hidden) * mask - # hack to handle LSTM - output = hidden[0] if isinstance(hidden, tuple) else hidden - - return hidden, output - - return forward - - -def StackedStep(layer, num_layers, lstm=False, dropout=0, train=True): - def forward(input, hidden, cells, mask): - assert (len(cells) == num_layers) - next_hidden = [] - - if lstm: - hidden = list(zip(*hidden)) - - for l in range(num_layers): - hy, output = layer(input, hidden[l], cells[l], mask) - next_hidden.append(hy) - input = output - - if dropout != 0 and l < num_layers - 1: - input = F.dropout(input, p=dropout, training=train, inplace=False) - - if lstm: - next_h, next_c = zip(*next_hidden) - next_hidden = ( - torch.cat(next_h, 0).view(num_layers, *next_h[0].size()), - torch.cat(next_c, 0).view(num_layers, *next_c[0].size()) - ) - else: - next_hidden = torch.cat(next_hidden, 0).view(num_layers, *next_hidden[0].size()) - - return next_hidden, input - - return forward - - -def AutogradMaskedStep(num_layers=1, dropout=0, train=True, lstm=False): - layer = MaskedStep() - - func = StackedStep(layer, - num_layers, - lstm=lstm, - dropout=dropout, - train=train) - - def forward(input, cells, hidden, mask): - nexth, output = func(input, hidden, cells, mask) - return output, nexth - - return forward - - -class MaskedRNNBase(nn.Module): - def __init__(self, Cell, input_size, hidden_size, - num_layers=1, bias=True, batch_first=False, - layer_dropout=0, step_dropout=0, bidirectional=False, initial_method = None , **kwargs): - """ - :param Cell: - :param input_size: - :param hidden_size: - :param num_layers: - :param bias: - :param batch_first: - :param layer_dropout: - :param step_dropout: - :param bidirectional: - :param kwargs: - """ - - super(MaskedRNNBase, self).__init__() - self.Cell = Cell - self.input_size = input_size - self.hidden_size = hidden_size - self.num_layers = num_layers - self.bias = bias - self.batch_first = batch_first - self.layer_dropout = layer_dropout - self.step_dropout = step_dropout - self.bidirectional = bidirectional - num_directions = 2 if bidirectional else 1 - - self.all_cells = [] - for layer in range(num_layers): # 初始化所有cell - for direction in range(num_directions): - layer_input_size = input_size if layer == 0 else hidden_size * num_directions - - cell = self.Cell(layer_input_size, hidden_size, self.bias, **kwargs) - self.all_cells.append(cell) - self.add_module('cell%d' % (layer * num_directions + direction), cell) # Max的代码写得真好看 - initial_parameter(self, initial_method) - def reset_parameters(self): - for cell in self.all_cells: - cell.reset_parameters() - - def forward(self, input, mask=None, hx=None): - batch_size = input.size(0) if self.batch_first else input.size(1) - lstm = self.Cell is nn.LSTMCell - if hx is None: - num_directions = 2 if self.bidirectional else 1 - hx = torch.autograd.Variable( - input.data.new(self.num_layers * num_directions, batch_size, self.hidden_size).zero_()) - if lstm: - hx = (hx, hx) - - func = AutogradMaskedRNN(num_layers=self.num_layers, - batch_first=self.batch_first, - step_dropout=self.step_dropout, - layer_dropout=self.layer_dropout, - train=self.training, - bidirectional=self.bidirectional, - lstm=lstm) # 传入all_cells，继续往底层封装走 - - output, hidden = func(input, self.all_cells, hx, - None if mask is None else mask.view(mask.size() + (1,))) # 这个+ (1, )是个什么操作？ - return output, hidden - - def step(self, input, hx=None, mask=None): - """Execute one step forward (only for one-directional RNN). - - :param Tensor input: input tensor of this step. (batch, input_size) - :param Tensor hx: the hidden state of last step. (num_layers, batch, hidden_size) - :param Tensor mask: the mask tensor of this step. (batch, ) - :returns: - **output** (batch, hidden_size), tensor containing the output of this step from the last layer of RNN. - **hn** (num_layers, batch, hidden_size), tensor containing the hidden state of this step - - """ - assert not self.bidirectional, "step only cannot be applied to bidirectional RNN." # aha, typo! - batch_size = input.size(0) - lstm = self.Cell is nn.LSTMCell - if hx is None: - hx = torch.autograd.Variable(input.data.new(self.num_layers, batch_size, self.hidden_size).zero_()) - if lstm: - hx = (hx, hx) - - func = AutogradMaskedStep(num_layers=self.num_layers, - dropout=self.step_dropout, - train=self.training, - lstm=lstm) - - output, hidden = func(input, self.all_cells, hx, mask) - return output, hidden - - -class MaskedRNN(MaskedRNNBase): - r"""Applies a multi-layer Elman RNN with costomized non-linearity to an - input sequence. - For each element in the input sequence, each layer computes the following - function. :math:`h_t = \tanh(w_{ih} * x_t + b_{ih} + w_{hh} * h_{(t-1)} + b_{hh})` - - where :math:`h_t` is the hidden state at time `t`, and :math:`x_t` is - the hidden state of the previous layer at time `t` or :math:`input_t` - for the first layer. If nonlinearity='relu', then `ReLU` is used instead - of `tanh`. - - - :param int input_size: The number of expected features in the input x - :param int hidden_size: The number of features in the hidden state h - :param int num_layers: Number of recurrent layers. - :param str nonlinearity: The non-linearity to use ['tanh'|'relu']. Default: 'tanh' - :param bool bias: If False, then the layer does not use bias weights b_ih and b_hh. Default: True - :param bool batch_first: If True, then the input and output tensors are provided as (batch, seq, feature) - :param float dropout: If non-zero, introduces a dropout layer on the outputs of each RNN layer except the last layer - :param bool bidirectional: If True, becomes a bidirectional RNN. Default: False - - Inputs: input, mask, h_0 - - **input** (seq_len, batch, input_size): tensor containing the features - of the input sequence. - **mask** (seq_len, batch): 0-1 tensor containing the mask of the input sequence. - - **h_0** (num_layers * num_directions, batch, hidden_size): tensor - containing the initial hidden state for each element in the batch. - Outputs: output, h_n - - **output** (seq_len, batch, hidden_size * num_directions): tensor - containing the output features (h_k) from the last layer of the RNN, - for each k. If a :class:`torch.nn.utils.rnn.PackedSequence` has - been given as the input, the output will also be a packed sequence. - - **h_n** (num_layers * num_directions, batch, hidden_size): tensor - containing the hidden state for k=seq_len. - """ - - def __init__(self, *args, **kwargs): - super(MaskedRNN, self).__init__(nn.RNNCell, *args, **kwargs) - - -class MaskedLSTM(MaskedRNNBase): - r"""Applies a multi-layer long short-term memory (LSTM) RNN to an input - sequence. - For each element in the input sequence, each layer computes the following - function. - - .. math:: - - \begin{array}{ll} - i_t = \mathrm{sigmoid}(W_{ii} x_t + b_{ii} + W_{hi} h_{(t-1)} + b_{hi}) \\ - f_t = \mathrm{sigmoid}(W_{if} x_t + b_{if} + W_{hf} h_{(t-1)} + b_{hf}) \\ - g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hc} h_{(t-1)} + b_{hg}) \\ - o_t = \mathrm{sigmoid}(W_{io} x_t + b_{io} + W_{ho} h_{(t-1)} + b_{ho}) \\ - c_t = f_t * c_{(t-1)} + i_t * g_t \\ - h_t = o_t * \tanh(c_t) - \end{array} - - where :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell - state at time `t`, :math:`x_t` is the hidden state of the previous layer at - time `t` or :math:`input_t` for the first layer, and :math:`i_t`, - :math:`f_t`, :math:`g_t`, :math:`o_t` are the input, forget, cell, - and out gates, respectively. - - :param int input_size: The number of expected features in the input x - :param int hidden_size: The number of features in the hidden state h - :param int num_layers: Number of recurrent layers. - :param bool bias: If False, then the layer does not use bias weights b_ih and b_hh. Default: True - :param bool batch_first: If True, then the input and output tensors are provided as (batch, seq, feature) - :param bool dropout: If non-zero, introduces a dropout layer on the outputs of each RNN layer except the last layer - :param bool bidirectional: If True, becomes a bidirectional RNN. Default: False - - Inputs: input, mask, (h_0, c_0) - - **input** (seq_len, batch, input_size): tensor containing the features - of the input sequence. - **mask** (seq_len, batch): 0-1 tensor containing the mask of the input sequence. - - **h_0** (num_layers \* num_directions, batch, hidden_size): tensor - containing the initial hidden state for each element in the batch. - - **c_0** (num_layers \* num_directions, batch, hidden_size): tensor - containing the initial cell state for each element in the batch. - Outputs: output, (h_n, c_n) - - **output** (seq_len, batch, hidden_size * num_directions): tensor - containing the output features `(h_t)` from the last layer of the RNN, - for each t. If a :class:`torch.nn.utils.rnn.PackedSequence` has been - given as the input, the output will also be a packed sequence. - - **h_n** (num_layers * num_directions, batch, hidden_size): tensor - containing the hidden state for t=seq_len - - **c_n** (num_layers * num_directions, batch, hidden_size): tensor - containing the cell state for t=seq_len - """ - - def __init__(self, *args, **kwargs): - super(MaskedLSTM, self).__init__(nn.LSTMCell, *args, **kwargs) - - -class MaskedGRU(MaskedRNNBase): - r"""Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence. - For each element in the input sequence, each layer computes the following - function: - - .. math:: - - \begin{array}{ll} - r_t = \mathrm{sigmoid}(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ - z_t = \mathrm{sigmoid}(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ - n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ - h_t = (1 - z_t) * n_t + z_t * h_{(t-1)} \\ - \end{array} - - where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the hidden - state of the previous layer at time `t` or :math:`input_t` for the first - layer, and :math:`r_t`, :math:`z_t`, :math:`n_t` are the reset, input, - and new gates, respectively. - - :param int input_size: The number of expected features in the input x - :param int hidden_size: The number of features in the hidden state h - :param int num_layers: Number of recurrent layers. - :param str nonlinearity: The non-linearity to use ['tanh'|'relu']. Default: 'tanh' - :param bool bias: If False, then the layer does not use bias weights b_ih and b_hh. Default: True - :param bool batch_first: If True, then the input and output tensors are provided as (batch, seq, feature) - :param bool dropout: If non-zero, introduces a dropout layer on the outputs of each RNN layer except the last layer - :param bool bidirectional: If True, becomes a bidirectional RNN. Default: False - - Inputs: input, mask, h_0 - - **input** (seq_len, batch, input_size): tensor containing the features - of the input sequence. - **mask** (seq_len, batch): 0-1 tensor containing the mask of the input sequence. - - **h_0** (num_layers * num_directions, batch, hidden_size): tensor - containing the initial hidden state for each element in the batch. - Outputs: output, h_n - - **output** (seq_len, batch, hidden_size * num_directions): tensor - containing the output features (h_k) from the last layer of the RNN, - for each k. If a :class:`torch.nn.utils.rnn.PackedSequence` has - been given as the input, the output will also be a packed sequence. - - **h_n** (num_layers * num_directions, batch, hidden_size): tensor - containing the hidden state for k=seq_len. - """ - - def __init__(self, *args, **kwargs): - super(MaskedGRU, self).__init__(nn.GRUCell, *args, **kwargs) diff --git a/fastNLP/modules/encoder/star_transformer.py b/fastNLP/modules/encoder/star_transformer.py new file mode 100644 index 00000000..1eec7c13 --- /dev/null +++ b/fastNLP/modules/encoder/star_transformer.py @@ -0,0 +1,162 @@ +""" +Star-Transformer 的encoder部分的 Pytorch 实现 +""" +__all__ = [ + "StarTransformer" +] + +import numpy as NP +import torch +from torch import nn +from torch.nn import functional as F + + +class StarTransformer(nn.Module): + """ + 别名：:class:`fastNLP.modules.StarTransformer` :class:`fastNLP.modules.encoder.star_transformer.StarTransformer` + + + Star-Transformer 的encoder部分。 输入3d的文本输入, 返回相同长度的文本编码 + + paper: https://arxiv.org/abs/1902.09113 + + :param int hidden_size: 输入维度的大小。同时也是输出维度的大小。 + :param int num_layers: star-transformer的层数 + :param int num_head: head的数量。 + :param int head_dim: 每个head的维度大小。 + :param float dropout: dropout 概率. Default: 0.1 + :param int max_len: int or None, 如果为int，输入序列的最大长度， + 模型会为输入序列加上position embedding。 + 若为`None`，忽略加上position embedding的步骤. Default: `None` + """ + + def __init__(self, hidden_size, num_layers, num_head, head_dim, dropout=0.1, max_len=None): + super(StarTransformer, self).__init__() + self.iters = num_layers + + self.norm = nn.ModuleList([nn.LayerNorm(hidden_size) for _ in range(self.iters)]) + self.ring_att = nn.ModuleList( + [_MSA1(hidden_size, nhead=num_head, head_dim=head_dim, dropout=dropout) + for _ in range(self.iters)]) + self.star_att = nn.ModuleList( + [_MSA2(hidden_size, nhead=num_head, head_dim=head_dim, dropout=dropout) + for _ in range(self.iters)]) + + if max_len is not None: + self.pos_emb = nn.Embedding(max_len, hidden_size) + else: + self.pos_emb = None + + def forward(self, data, mask): + """ + :param FloatTensor data: [batch, length, hidden] 输入的序列 + :param ByteTensor mask: [batch, length] 输入序列的padding mask, 在没有内容(padding 部分) 为 0, + 否则为 1 + :return: [batch, length, hidden] 编码后的输出序列 + + [batch, hidden] 全局 relay 节点, 详见论文 + """ + + def norm_func(f, x): + # B, H, L, 1 + return f(x.permute(0, 2, 3, 1)).permute(0, 3, 1, 2) + + B, L, H = data.size() + mask = (mask == 0) # flip the mask for masked_fill_ + smask = torch.cat([torch.zeros(B, 1, ).byte().to(mask), mask], 1) + + embs = data.permute(0, 2, 1)[:, :, :, None] # B H L 1 + if self.pos_emb: + P = self.pos_emb(torch.arange(L, dtype=torch.long, device=embs.device) \ + .view(1, L)).permute(0, 2, 1).contiguous()[:, :, :, None] # 1 H L 1 + embs = embs + P + + nodes = embs + relay = embs.mean(2, keepdim=True) + ex_mask = mask[:, None, :, None].expand(B, H, L, 1) + r_embs = embs.view(B, H, 1, L) + for i in range(self.iters): + ax = torch.cat([r_embs, relay.expand(B, H, 1, L)], 2) + nodes = nodes + F.leaky_relu(self.ring_att[i](norm_func(self.norm[i], nodes), ax=ax)) + relay = F.leaky_relu(self.star_att[i](relay, torch.cat([relay, nodes], 2), smask)) + + nodes = nodes.masked_fill_(ex_mask, 0) + + nodes = nodes.view(B, H, L).permute(0, 2, 1) + + return nodes, relay.view(B, H) + + +class _MSA1(nn.Module): + def __init__(self, nhid, nhead=10, head_dim=10, dropout=0.1): + super(_MSA1, self).__init__() + # Multi-head Self Attention Case 1, doing self-attention for small regions + # Due to the architecture of GPU, using hadamard production and summation are faster than dot production when unfold_size is very small + self.WQ = nn.Conv2d(nhid, nhead * head_dim, 1) + self.WK = nn.Conv2d(nhid, nhead * head_dim, 1) + self.WV = nn.Conv2d(nhid, nhead * head_dim, 1) + self.WO = nn.Conv2d(nhead * head_dim, nhid, 1) + + self.drop = nn.Dropout(dropout) + + # print('NUM_HEAD', nhead, 'DIM_HEAD', head_dim) + self.nhid, self.nhead, self.head_dim, self.unfold_size = nhid, nhead, head_dim, 3 + + def forward(self, x, ax=None): + # x: B, H, L, 1, ax : B, H, X, L append features + nhid, nhead, head_dim, unfold_size = self.nhid, self.nhead, self.head_dim, self.unfold_size + B, H, L, _ = x.shape + + q, k, v = self.WQ(x), self.WK(x), self.WV(x) # x: (B,H,L,1) + + if ax is not None: + aL = ax.shape[2] + ak = self.WK(ax).view(B, nhead, head_dim, aL, L) + av = self.WV(ax).view(B, nhead, head_dim, aL, L) + q = q.view(B, nhead, head_dim, 1, L) + k = F.unfold(k.view(B, nhead * head_dim, L, 1), (unfold_size, 1), padding=(unfold_size // 2, 0)) \ + .view(B, nhead, head_dim, unfold_size, L) + v = F.unfold(v.view(B, nhead * head_dim, L, 1), (unfold_size, 1), padding=(unfold_size // 2, 0)) \ + .view(B, nhead, head_dim, unfold_size, L) + if ax is not None: + k = torch.cat([k, ak], 3) + v = torch.cat([v, av], 3) + + alphas = self.drop(F.softmax((q * k).sum(2, keepdim=True) / NP.sqrt(head_dim), 3)) # B N L 1 U + att = (alphas * v).sum(3).view(B, nhead * head_dim, L, 1) + + ret = self.WO(att) + + return ret + + +class _MSA2(nn.Module): + def __init__(self, nhid, nhead=10, head_dim=10, dropout=0.1): + # Multi-head Self Attention Case 2, a broadcastable query for a sequence key and value + super(_MSA2, self).__init__() + self.WQ = nn.Conv2d(nhid, nhead * head_dim, 1) + self.WK = nn.Conv2d(nhid, nhead * head_dim, 1) + self.WV = nn.Conv2d(nhid, nhead * head_dim, 1) + self.WO = nn.Conv2d(nhead * head_dim, nhid, 1) + + self.drop = nn.Dropout(dropout) + + # print('NUM_HEAD', nhead, 'DIM_HEAD', head_dim) + self.nhid, self.nhead, self.head_dim, self.unfold_size = nhid, nhead, head_dim, 3 + + def forward(self, x, y, mask=None): + # x: B, H, 1, 1, 1 y: B H L 1 + nhid, nhead, head_dim, unfold_size = self.nhid, self.nhead, self.head_dim, self.unfold_size + B, H, L, _ = y.shape + + q, k, v = self.WQ(x), self.WK(y), self.WV(y) + + q = q.view(B, nhead, 1, head_dim) # B, H, 1, 1 -> B, N, 1, h + k = k.view(B, nhead, head_dim, L) # B, H, L, 1 -> B, N, h, L + v = v.view(B, nhead, head_dim, L).permute(0, 1, 3, 2) # B, H, L, 1 -> B, N, L, h + pre_a = torch.matmul(q, k) / NP.sqrt(head_dim) + if mask is not None: + pre_a = pre_a.masked_fill(mask[:, None, None, :], -float('inf')) + alphas = self.drop(F.softmax(pre_a, 3)) # B, N, 1, L + att = torch.matmul(alphas, v).view(B, -1, 1, 1) # B, N, 1, h -> B, N*h, 1, 1 + return self.WO(att) diff --git a/fastNLP/modules/encoder/transformer.py b/fastNLP/modules/encoder/transformer.py index 615a6f34..698ff95c 100644 --- a/fastNLP/modules/encoder/transformer.py +++ b/fastNLP/modules/encoder/transformer.py @@ -1,29 +1,71 @@ +__all__ = [ + "TransformerEncoder" +] from torch import nn -from ..aggregator.attention import MultiHeadAtte -from ..other_modules import LayerNormalization +from ..aggregator.attention import MultiHeadAttention +from ..dropout import TimestepDropout class TransformerEncoder(nn.Module): + """ + 别名：:class:`fastNLP.modules.TransformerEncoder` :class:`fastNLP.modules.encoder.transformer.TransformerEncoder` + + + transformer的encoder模块，不包含embedding层 + + :param int num_layers: transformer的层数 + :param int model_size: 输入维度的大小。同时也是输出维度的大小。 + :param int inner_size: FFN层的hidden大小 + :param int key_size: 每个head的维度大小。 + :param int value_size: 每个head中value的维度。 + :param int num_head: head的数量。 + :param float dropout: dropout概率. Default: 0.1 + """ + class SubLayer(nn.Module): - def __init__(self, input_size, output_size, key_size, value_size, num_atte): + def __init__(self, model_size, inner_size, key_size, value_size, num_head, dropout=0.1): super(TransformerEncoder.SubLayer, self).__init__() - self.atte = MultiHeadAtte(input_size, output_size, key_size, value_size, num_atte) - self.norm1 = LayerNormalization(output_size) - self.ffn = nn.Sequential(nn.Linear(output_size, output_size), + self.atte = MultiHeadAttention(model_size, key_size, value_size, num_head, dropout) + self.norm1 = nn.LayerNorm(model_size) + self.ffn = nn.Sequential(nn.Linear(model_size, inner_size), nn.ReLU(), - nn.Linear(output_size, output_size)) - self.norm2 = LayerNormalization(output_size) + nn.Linear(inner_size, model_size), + TimestepDropout(dropout), ) + self.norm2 = nn.LayerNorm(model_size) + + def forward(self, input, seq_mask=None, atte_mask_out=None): + """ - def forward(self, input, seq_mask): - attention = self.atte(input) + :param input: [batch, seq_len, model_size] + :param seq_mask: [batch, seq_len] + :return: [batch, seq_len, model_size] + """ + attention = self.atte(input, input, input, atte_mask_out) norm_atte = self.norm1(attention + input) + attention *= seq_mask output = self.ffn(norm_atte) - return self.norm2(output + norm_atte) - + output = self.norm2(output + norm_atte) + output *= seq_mask + return output + def __init__(self, num_layers, **kargs): super(TransformerEncoder, self).__init__() - self.layers = nn.Sequential(*[self.SubLayer(**kargs) for _ in range(num_layers)]) - + self.layers = nn.ModuleList([self.SubLayer(**kargs) for _ in range(num_layers)]) + def forward(self, x, seq_mask=None): - return self.layers(x, seq_mask) + """ + :param x: [batch, seq_len, model_size] 输入序列 + :param seq_mask: [batch, seq_len] 输入序列的padding mask, 若为 ``None`` , 生成全1向量. + Default: ``None`` + :return: [batch, seq_len, model_size] 输出序列 + """ + output = x + if seq_mask is None: + atte_mask_out = None + else: + atte_mask_out = (seq_mask < 1)[:, None, :] + seq_mask = seq_mask[:, :, None] + for layer in self.layers: + output = layer(output, seq_mask, atte_mask_out) + return output diff --git a/fastNLP/modules/encoder/variational_rnn.py b/fastNLP/modules/encoder/variational_rnn.py index a7902813..29b728e5 100644 --- a/fastNLP/modules/encoder/variational_rnn.py +++ b/fastNLP/modules/encoder/variational_rnn.py @@ -1,7 +1,15 @@ +""" +Variational RNN 的 Pytorch 实现 +""" +__all__ = [ + "VarRNN", + "VarLSTM", + "VarGRU" +] + import torch import torch.nn as nn from torch.nn.utils.rnn import PackedSequence, pack_padded_sequence, pad_packed_sequence -from fastNLP.modules.utils import initial_parameter try: from torch import flip @@ -9,39 +17,45 @@ except ImportError: def flip(x, dims): indices = [slice(None)] * x.dim() for dim in dims: - indices[dim] = torch.arange(x.size(dim) - 1, -1, -1, dtype=torch.long, device=x.device) + indices[dim] = torch.arange( + x.size(dim) - 1, -1, -1, dtype=torch.long, device=x.device) return x[tuple(indices)] +from ..utils import initial_parameter + class VarRnnCellWrapper(nn.Module): - """Wrapper for normal RNN Cells, make it support variational dropout """ - + Wrapper for normal RNN Cells, make it support variational dropout + """ + def __init__(self, cell, hidden_size, input_p, hidden_p): super(VarRnnCellWrapper, self).__init__() self.cell = cell self.hidden_size = hidden_size self.input_p = input_p self.hidden_p = hidden_p - + def forward(self, input_x, hidden, mask_x, mask_h, is_reversed=False): """ :param PackedSequence input_x: [seq_len, batch_size, input_size] :param hidden: for LSTM, tuple of (h_0, c_0), [batch_size, hidden_size] - for other RNN, h_0, [batch_size, hidden_size] + for other RNN, h_0, [batch_size, hidden_size] :param mask_x: [batch_size, input_size] dropout mask for input :param mask_h: [batch_size, hidden_size] dropout mask for hidden :return PackedSequence output: [seq_len, bacth_size, hidden_size] hidden: for LSTM, tuple of (h_n, c_n), [batch_size, hidden_size] for other RNN, h_n, [batch_size, hidden_size] """ + def get_hi(hi, h0, size): h0_size = size - hi.size(0) if h0_size > 0: return torch.cat([hi, h0[:h0_size]], dim=0) return hi[:size] + is_lstm = isinstance(hidden, tuple) - input, batch_sizes = input_x + input, batch_sizes = input_x.data, input_x.batch_sizes output = [] cell = self.cell if is_reversed: @@ -50,7 +64,7 @@ class VarRnnCellWrapper(nn.Module): else: batch_iter = batch_sizes idx = 0 - + if is_lstm: hn = (hidden[0].clone(), hidden[1].clone()) else: @@ -58,15 +72,16 @@ class VarRnnCellWrapper(nn.Module): hi = hidden for size in batch_iter: if is_reversed: - input_i = input[idx-size: idx] * mask_x[:size] + input_i = input[idx - size: idx] * mask_x[:size] idx -= size else: - input_i = input[idx: idx+size] * mask_x[:size] + input_i = input[idx: idx + size] * mask_x[:size] idx += size mask_hi = mask_h[:size] if is_lstm: hx, cx = hi - hi = (get_hi(hx, hidden[0], size) * mask_hi, get_hi(cx, hidden[1], size)) + hi = (get_hi(hx, hidden[0], size) * + mask_hi, get_hi(cx, hidden[1], size)) hi = cell(input_i, hi) hn[0][:size] = hi[0] hn[1][:size] = hi[1] @@ -76,7 +91,7 @@ class VarRnnCellWrapper(nn.Module): hi = cell(input_i, hi) hn[:size] = hi output.append(hi) - + if is_reversed: output = list(reversed(output)) output = torch.cat(output, dim=0) @@ -84,11 +99,25 @@ class VarRnnCellWrapper(nn.Module): class VarRNNBase(nn.Module): - """Implementation of Variational Dropout RNN network. - refer to `A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (Yarin Gal and Zoubin Ghahramani, 2016) - https://arxiv.org/abs/1512.05287`. """ + Variational Dropout RNN 实现. + + 论文参考: `A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (Yarin Gal and Zoubin Ghahramani, 2016) + https://arxiv.org/abs/1512.05287`. + :param mode: rnn 模式, (lstm or not) + :param Cell: rnn cell 类型, (lstm, gru, etc) + :param input_size: 输入 `x` 的特征维度 + :param hidden_size: 隐状态 `h` 的特征维度 + :param num_layers: rnn的层数. Default: 1 + :param bias: 如果为 ``False``, 模型将不会使用bias. Default: ``True`` + :param batch_first: 若为 ``True``, 输入和输出 ``Tensor`` 形状为 + (batch, seq, feature). Default: ``False`` + :param input_dropout: 对输入的dropout概率. Default: 0 + :param hidden_dropout: 对每个隐状态的dropout概率. Default: 0 + :param bidirectional: 若为 ``True``, 使用双向的RNN. Default: ``False`` + """ + def __init__(self, mode, Cell, input_size, hidden_size, num_layers=1, bias=True, batch_first=False, input_dropout=0, hidden_dropout=0, bidirectional=False): @@ -108,49 +137,65 @@ class VarRNNBase(nn.Module): for direction in range(self.num_directions): input_size = self.input_size if layer == 0 else self.hidden_size * self.num_directions cell = Cell(input_size, self.hidden_size, bias) - self._all_cells.append(VarRnnCellWrapper(cell, self.hidden_size, input_dropout, hidden_dropout)) + self._all_cells.append(VarRnnCellWrapper( + cell, self.hidden_size, input_dropout, hidden_dropout)) initial_parameter(self) self.is_lstm = (self.mode == "LSTM") - + def _forward_one(self, n_layer, n_direction, input, hx, mask_x, mask_h): is_lstm = self.is_lstm idx = self.num_directions * n_layer + n_direction cell = self._all_cells[idx] hi = (hx[0][idx], hx[1][idx]) if is_lstm else hx[idx] - output_x, hidden_x = cell(input, hi, mask_x, mask_h, is_reversed=(n_direction == 1)) + output_x, hidden_x = cell( + input, hi, mask_x, mask_h, is_reversed=(n_direction == 1)) return output_x, hidden_x + + def forward(self, x, hx=None): + """ - def forward(self, input, hx=None): + :param x: [batch, seq_len, input_size] 输入序列 + :param hx: [batch, hidden_size] 初始隐状态, 若为 ``None`` , 设为全1向量. Default: ``None`` + :return (output, ht): [batch, seq_len, hidden_size*num_direction] 输出序列 + 和 [batch, hidden_size*num_direction] 最后时刻隐状态 + """ is_lstm = self.is_lstm - is_packed = isinstance(input, PackedSequence) + is_packed = isinstance(x, PackedSequence) if not is_packed: - seq_len = input.size(1) if self.batch_first else input.size(0) - max_batch_size = input.size(0) if self.batch_first else input.size(1) - seq_lens = torch.LongTensor([seq_len for _ in range(max_batch_size)]) - input, batch_sizes = pack_padded_sequence(input, seq_lens, batch_first=self.batch_first) + seq_len = x.size(1) if self.batch_first else x.size(0) + max_batch_size = x.size(0) if self.batch_first else x.size(1) + seq_lens = torch.LongTensor( + [seq_len for _ in range(max_batch_size)]) + x = pack_padded_sequence(x, seq_lens, batch_first=self.batch_first) else: - max_batch_size = int(input.batch_sizes[0]) - input, batch_sizes = input - + max_batch_size = int(x.batch_sizes[0]) + x, batch_sizes = x.data, x.batch_sizes + if hx is None: - hx = input.new_zeros(self.num_layers * self.num_directions, - max_batch_size, self.hidden_size, requires_grad=True) + hx = x.new_zeros(self.num_layers * self.num_directions, + max_batch_size, self.hidden_size, requires_grad=True) if is_lstm: hx = (hx, hx.new_zeros(hx.size(), requires_grad=True)) - - mask_x = input.new_ones((max_batch_size, self.input_size)) - mask_out = input.new_ones((max_batch_size, self.hidden_size * self.num_directions)) - mask_h_ones = input.new_ones((max_batch_size, self.hidden_size)) - nn.functional.dropout(mask_x, p=self.input_dropout, training=self.training, inplace=True) - nn.functional.dropout(mask_out, p=self.hidden_dropout, training=self.training, inplace=True) - - hidden = input.new_zeros((self.num_layers*self.num_directions, max_batch_size, self.hidden_size)) + + mask_x = x.new_ones((max_batch_size, self.input_size)) + mask_out = x.new_ones( + (max_batch_size, self.hidden_size * self.num_directions)) + mask_h_ones = x.new_ones((max_batch_size, self.hidden_size)) + nn.functional.dropout(mask_x, p=self.input_dropout, + training=self.training, inplace=True) + nn.functional.dropout(mask_out, p=self.hidden_dropout, + training=self.training, inplace=True) + + hidden = x.new_zeros( + (self.num_layers * self.num_directions, max_batch_size, self.hidden_size)) if is_lstm: - cellstate = input.new_zeros((self.num_layers*self.num_directions, max_batch_size, self.hidden_size)) + cellstate = x.new_zeros( + (self.num_layers * self.num_directions, max_batch_size, self.hidden_size)) for layer in range(self.num_layers): output_list = [] - input_seq = PackedSequence(input, batch_sizes) - mask_h = nn.functional.dropout(mask_h_ones, p=self.hidden_dropout, training=self.training, inplace=False) + input_seq = PackedSequence(x, batch_sizes) + mask_h = nn.functional.dropout( + mask_h_ones, p=self.hidden_dropout, training=self.training, inplace=False) for direction in range(self.num_directions): output_x, hidden_x = self._forward_one(layer, direction, input_seq, hx, mask_x if layer == 0 else mask_out, mask_h) @@ -161,72 +206,90 @@ class VarRNNBase(nn.Module): cellstate[idx] = hidden_x[1] else: hidden[idx] = hidden_x - input = torch.cat(output_list, dim=-1) - + x = torch.cat(output_list, dim=-1) + if is_lstm: hidden = (hidden, cellstate) - + if is_packed: - output = PackedSequence(input, batch_sizes) + output = PackedSequence(x, batch_sizes) else: - input = PackedSequence(input, batch_sizes) - output, _ = pad_packed_sequence(input, batch_first=self.batch_first) - + x = PackedSequence(x, batch_sizes) + output, _ = pad_packed_sequence(x, batch_first=self.batch_first) + return output, hidden class VarLSTM(VarRNNBase): - """Variational Dropout LSTM. """ + 别名：:class:`fastNLP.modules.VarLSTM` :class:`fastNLP.modules.encoder.variational_rnn.VarLSTM` + Variational Dropout LSTM. + + :param input_size: 输入 `x` 的特征维度 + :param hidden_size: 隐状态 `h` 的特征维度 + :param num_layers: rnn的层数. Default: 1 + :param bias: 如果为 ``False``, 模型将不会使用bias. Default: ``True`` + :param batch_first: 若为 ``True``, 输入和输出 ``Tensor`` 形状为 + (batch, seq, feature). Default: ``False`` + :param input_dropout: 对输入的dropout概率. Default: 0 + :param hidden_dropout: 对每个隐状态的dropout概率. Default: 0 + :param bidirectional: 若为 ``True``, 使用双向的LSTM. Default: ``False`` + """ + def __init__(self, *args, **kwargs): - super(VarLSTM, self).__init__(mode="LSTM", Cell=nn.LSTMCell, *args, **kwargs) + super(VarLSTM, self).__init__( + mode="LSTM", Cell=nn.LSTMCell, *args, **kwargs) + + def forward(self, x, hx=None): + return super(VarLSTM, self).forward(x, hx) class VarRNN(VarRNNBase): - """Variational Dropout RNN. """ + 别名：:class:`fastNLP.modules.VarRNN` :class:`fastNLP.modules.encoder.variational_rnn.VarRNN` + + Variational Dropout RNN. + :param input_size: 输入 `x` 的特征维度 + :param hidden_size: 隐状态 `h` 的特征维度 + :param num_layers: rnn的层数. Default: 1 + :param bias: 如果为 ``False``, 模型将不会使用bias. Default: ``True`` + :param batch_first: 若为 ``True``, 输入和输出 ``Tensor`` 形状为 + (batch, seq, feature). Default: ``False`` + :param input_dropout: 对输入的dropout概率. Default: 0 + :param hidden_dropout: 对每个隐状态的dropout概率. Default: 0 + :param bidirectional: 若为 ``True``, 使用双向的RNN. Default: ``False`` + """ + def __init__(self, *args, **kwargs): - super(VarRNN, self).__init__(mode="RNN", Cell=nn.RNNCell, *args, **kwargs) + super(VarRNN, self).__init__( + mode="RNN", Cell=nn.RNNCell, *args, **kwargs) + + def forward(self, x, hx=None): + return super(VarRNN, self).forward(x, hx) class VarGRU(VarRNNBase): - """Variational Dropout GRU. """ + 别名：:class:`fastNLP.modules.VarGRU` :class:`fastNLP.modules.encoder.variational_rnn.VarGRU` + + Variational Dropout GRU. + :param input_size: 输入 `x` 的特征维度 + :param hidden_size: 隐状态 `h` 的特征维度 + :param num_layers: rnn的层数. Default: 1 + :param bias: 如果为 ``False``, 模型将不会使用bias. Default: ``True`` + :param batch_first: 若为 ``True``, 输入和输出 ``Tensor`` 形状为 + (batch, seq, feature). Default: ``False`` + :param input_dropout: 对输入的dropout概率. Default: 0 + :param hidden_dropout: 对每个隐状态的dropout概率. Default: 0 + :param bidirectional: 若为 ``True``, 使用双向的GRU. Default: ``False`` + """ + def __init__(self, *args, **kwargs): - super(VarGRU, self).__init__(mode="GRU", Cell=nn.GRUCell, *args, **kwargs) - -# if __name__ == '__main__': -# x = torch.Tensor([[1,2,3], [4,5,0], [6,0,0]])[:,:,None] * 0.1 -# mask = (x != 0).float().view(3, -1) -# seq_lens = torch.LongTensor([3,2,1]) -# y = torch.Tensor([[0,1,1], [1,1,0], [0,0,0]]) -# # rev = _reverse_packed_sequence(pack) -# # # print(rev) -# lstm = VarLSTM(input_size=1, num_layers=2, hidden_size=2, -# batch_first=True, bidirectional=True, -# input_dropout=0.0, hidden_dropout=0.0,) -# # lstm = nn.LSTM(input_size=1, num_layers=2, hidden_size=2, -# # batch_first=True, bidirectional=True,) -# loss = nn.BCELoss() -# m = nn.Sigmoid() -# optim = torch.optim.SGD(lstm.parameters(), lr=1e-3) -# for i in range(2000): -# optim.zero_grad() -# pack = pack_padded_sequence(x, seq_lens, batch_first=True) -# out, hidden = lstm(pack) -# out, lens = pad_packed_sequence(out, batch_first=True) -# # print(lens) -# # print(out) -# # print(hidden[0]) -# # print(hidden[0].size()) -# # print(hidden[1]) -# out = out.sum(-1) -# out = m(out) * mask -# l = loss(out, y) -# l.backward() -# optim.step() -# if i % 50 == 0: -# print(out) + super(VarGRU, self).__init__( + mode="GRU", Cell=nn.GRUCell, *args, **kwargs) + + def forward(self, x, hx=None): + return super(VarGRU, self).forward(x, hx) diff --git a/fastNLP/modules/other_modules.py b/fastNLP/modules/other_modules.py deleted file mode 100644 index 92a08ba1..00000000 --- a/fastNLP/modules/other_modules.py +++ /dev/null @@ -1,186 +0,0 @@ -import numpy as np -import torch -import torch.nn as nn -import torch.nn.functional as F -import torch.utils.data -from torch.nn import Parameter - - -class GroupNorm(nn.Module): - def __init__(self, num_features, num_groups=20, eps=1e-5): - super(GroupNorm, self).__init__() - self.weight = nn.Parameter(torch.ones(1, num_features, 1)) - self.bias = nn.Parameter(torch.zeros(1, num_features, 1)) - self.num_groups = num_groups - self.eps = eps - - def forward(self, x): - N, C, H = x.size() - G = self.num_groups - assert C % G == 0 - - x = x.view(N, G, -1) - mean = x.mean(-1, keepdim=True) - var = x.var(-1, keepdim=True) - - x = (x - mean) / (var + self.eps).sqrt() - x = x.view(N, C, H) - return x * self.weight + self.bias - - -class LayerNormalization(nn.Module): - """ - - :param int layer_size: - :param float eps: default=1e-3 - """ - def __init__(self, layer_size, eps=1e-3): - super(LayerNormalization, self).__init__() - - self.eps = eps - self.a_2 = nn.Parameter(torch.ones(1, layer_size, requires_grad=True)) - self.b_2 = nn.Parameter(torch.zeros(1, layer_size, requires_grad=True)) - - def forward(self, z): - if z.size(1) == 1: - return z - - mu = torch.mean(z, keepdim=True, dim=-1) - sigma = torch.std(z, keepdim=True, dim=-1) - ln_out = (z - mu) / (sigma + self.eps) - ln_out = ln_out * self.a_2 + self.b_2 - return ln_out - - -class BiLinear(nn.Module): - def __init__(self, n_left, n_right, n_out, bias=True): - """ - - :param int n_left: size of left input - :param int n_right: size of right input - :param int n_out: size of output - :param bool bias: If set to False, the layer will not learn an additive bias. Default: True - """ - super(BiLinear, self).__init__() - self.n_left = n_left - self.n_right = n_right - self.n_out = n_out - - self.U = Parameter(torch.Tensor(self.n_out, self.n_left, self.n_right)) - self.W_l = Parameter(torch.Tensor(self.n_out, self.n_left)) - self.W_r = Parameter(torch.Tensor(self.n_out, self.n_left)) - - if bias: - self.bias = Parameter(torch.Tensor(n_out)) - else: - self.register_parameter('bias', None) - - self.reset_parameters() - - def reset_parameters(self): - nn.init.xavier_uniform_(self.W_l) - nn.init.xavier_uniform_(self.W_r) - nn.init.constant_(self.bias, 0.) - nn.init.xavier_uniform_(self.U) - - def forward(self, input_left, input_right): - """ - :param Tensor input_left: the left input tensor with shape = [batch1, batch2, ..., left_features] - :param Tensor input_right: the right input tensor with shape = [batch1, batch2, ..., right_features] - - """ - left_size = input_left.size() - right_size = input_right.size() - assert left_size[:-1] == right_size[:-1], \ - "batch size of left and right inputs mis-match: (%s, %s)" % (left_size[:-1], right_size[:-1]) - batch = int(np.prod(left_size[:-1])) - - # convert left and right input to matrices [batch, left_features], [batch, right_features] - input_left = input_left.view(batch, self.n_left) - input_right = input_right.view(batch, self.n_right) - - # output [batch, out_features] - output = F.bilinear(input_left, input_right, self.U, self.bias) - output = output + \ - F.linear(input_left, self.W_l, None) + \ - F.linear(input_right, self.W_r, None) - # convert back to [batch1, batch2, ..., out_features] - return output.view(left_size[:-1] + (self.n_out,)) - - def __repr__(self): - return self.__class__.__name__ + ' (' \ - + 'in1_features=' + str(self.n_left) \ - + ', in2_features=' + str(self.n_right) \ - + ', out_features=' + str(self.n_out) + ')' - - -class BiAffine(nn.Module): - def __init__(self, n_enc, n_dec, n_labels, biaffine=True, **kwargs): - """ - - :param int n_enc: the dimension of the encoder input. - :param int n_dec: the dimension of the decoder input. - :param int n_labels: the number of labels of the crf layer - :param bool biaffine: if apply bi-affine parameter. - """ - super(BiAffine, self).__init__() - self.n_enc = n_enc - self.n_dec = n_dec - self.num_labels = n_labels - self.biaffine = biaffine - - self.W_d = Parameter(torch.Tensor(self.num_labels, self.n_dec)) - self.W_e = Parameter(torch.Tensor(self.num_labels, self.n_enc)) - self.b = Parameter(torch.Tensor(self.num_labels, 1, 1)) - if self.biaffine: - self.U = Parameter(torch.Tensor(self.num_labels, self.n_dec, self.n_enc)) - else: - self.register_parameter('U', None) - - self.reset_parameters() - - def reset_parameters(self): - nn.init.xavier_uniform_(self.W_d) - nn.init.xavier_uniform_(self.W_e) - nn.init.constant_(self.b, 0.) - if self.biaffine: - nn.init.xavier_uniform_(self.U) - - def forward(self, input_d, input_e, mask_d=None, mask_e=None): - """ - - :param Tensor input_d: the decoder input tensor with shape = [batch, length_decoder, input_size] - :param Tensor input_e: the child input tensor with shape = [batch, length_encoder, input_size] - :param mask_d: Tensor or None, the mask tensor for decoder with shape = [batch, length_decoder] - :param mask_e: Tensor or None, the mask tensor for encoder with shape = [batch, length_encoder] - :returns: Tensor, the energy tensor with shape = [batch, num_label, length, length] - """ - assert input_d.size(0) == input_e.size(0), 'batch sizes of encoder and decoder are requires to be equal.' - batch, length_decoder, _ = input_d.size() - _, length_encoder, _ = input_e.size() - - # compute decoder part: [num_label, input_size_decoder] * [batch, input_size_decoder, length_decoder] - # the output shape is [batch, num_label, length_decoder] - out_d = torch.matmul(self.W_d, input_d.transpose(1, 2)).unsqueeze(3) - # compute decoder part: [num_label, input_size_encoder] * [batch, input_size_encoder, length_encoder] - # the output shape is [batch, num_label, length_encoder] - out_e = torch.matmul(self.W_e, input_e.transpose(1, 2)).unsqueeze(2) - - # output shape [batch, num_label, length_decoder, length_encoder] - if self.biaffine: - # compute bi-affine part - # [batch, 1, length_decoder, input_size_decoder] * [num_labels, input_size_decoder, input_size_encoder] - # output shape [batch, num_label, length_decoder, input_size_encoder] - output = torch.matmul(input_d.unsqueeze(1), self.U) - # [batch, num_label, length_decoder, input_size_encoder] * [batch, 1, input_size_encoder, length_encoder] - # output shape [batch, num_label, length_decoder, length_encoder] - output = torch.matmul(output, input_e.unsqueeze(1).transpose(2, 3)) - - output = output + out_d + out_e + self.b - else: - output = out_d + out_d + self.b - - if mask_d is not None: - output = output * mask_d.unsqueeze(1).unsqueeze(3) * mask_e.unsqueeze(1).unsqueeze(2) - - return output diff --git a/fastNLP/modules/utils.py b/fastNLP/modules/utils.py index 5287bca4..741429bb 100644 --- a/fastNLP/modules/utils.py +++ b/fastNLP/modules/utils.py @@ -1,16 +1,11 @@ +from functools import reduce + +import numpy as np import torch import torch.nn as nn import torch.nn.init as init -def mask_softmax(matrix, mask): - if mask is None: - result = torch.nn.functional.softmax(matrix, dim=-1) - else: - raise NotImplementedError - return result - - def initial_parameter(net, initial_method=None): """A method used to initialize the weights of PyTorch models. @@ -60,7 +55,8 @@ def initial_parameter(net, initial_method=None): init_method(w.data) # weight else: init.normal_(w.data) # bias - elif hasattr(m, 'weight') and m.weight.requires_grad: + elif m is not None and hasattr(m, 'weight') and \ + hasattr(m.weight, "requires_grad"): init_method(m.weight.data) else: for w in m.parameters(): @@ -74,16 +70,63 @@ def initial_parameter(net, initial_method=None): net.apply(weights_init) -def seq_mask(seq_len, max_len): - """Create sequence mask. +def get_embeddings(init_embed): + """ + 根据输入的init_embed生成nn.Embedding对象。 + + :param init_embed: 可以是 tuple:(num_embedings, embedding_dim), 即embedding的大小和每个词的维度;也可以传入 + nn.Embedding 对象, 此时就以传入的对象作为embedding; 传入np.ndarray也行，将使用传入的ndarray作为作为Embedding初始 + 化; 传入orch.Tensor, 将使用传入的值作为Embedding初始化。 + :return nn.Embedding embeddings: + """ + if isinstance(init_embed, tuple): + res = nn.Embedding( + num_embeddings=init_embed[0], embedding_dim=init_embed[1]) + elif isinstance(init_embed, nn.Embedding): + res = init_embed + elif isinstance(init_embed, torch.Tensor): + res = nn.Embedding.from_pretrained(init_embed, freeze=False) + elif isinstance(init_embed, np.ndarray): + init_embed = torch.tensor(init_embed, dtype=torch.float32) + res = nn.Embedding.from_pretrained(init_embed, freeze=False) + else: + raise TypeError( + 'invalid init_embed type: {}'.format((type(init_embed)))) + return res + - :param seq_len: list or torch.Tensor, the lengths of sequences in a batch. - :param max_len: int, the maximum sequence length in a batch. - :return: mask, torch.LongTensor, [batch_size, max_len] +def summary(model: nn.Module): + """ + 得到模型的总参数量 + :params model: Pytorch 模型 + :return tuple: 包含总参数量，可训练参数量，不可训练参数量 """ - if not isinstance(seq_len, torch.Tensor): - seq_len = torch.LongTensor(seq_len) - seq_len = seq_len.view(-1, 1).long() # [batch_size, 1] - seq_range = torch.arange(start=0, end=max_len, dtype=torch.long, device=seq_len.device).view(1, -1) # [1, max_len] - return torch.gt(seq_len, seq_range) # [batch_size, max_len] + train = [] + nontrain = [] + + def layer_summary(module: nn.Module): + def count_size(sizes): + return reduce(lambda x, y: x*y, sizes) + + for p in module.parameters(recurse=False): + if p.requires_grad: + train.append(count_size(p.shape)) + else: + nontrain.append(count_size(p.shape)) + for subm in module.children(): + layer_summary(subm) + + layer_summary(model) + total_train = sum(train) + total_nontrain = sum(nontrain) + total = total_train + total_nontrain + strings = [] + strings.append('Total params: {:,}'.format(total)) + strings.append('Trainable params: {:,}'.format(total_train)) + strings.append('Non-trainable params: {:,}'.format(total_nontrain)) + max_len = len(max(strings, key=len)) + bar = '-'*(max_len + 3) + strings = [bar] + strings + [bar] + print('\n'.join(strings)) + return total, total_train, total_nontrain diff --git a/legacy/api/README.md b/legacy/api/README.md new file mode 100644 index 00000000..73560f9f --- /dev/null +++ b/legacy/api/README.md @@ -0,0 +1,44 @@ +# fastNLP 高级接口 + +### 环境与配置 +1. 系统环境：linux/ubuntu(推荐) +2. 编程语言：Python>=3.6 +3. Python包依赖 + - **torch==1.0** + - numpy>=1.14.2 + +### 中文分词 +```python +text = ['编者按：7月12日，英国航空航天系统公司公布了该公司研制的第一款高科技隐形无人机雷电之神。', + '这款飞行从外型上来看酷似电影中的太空飞行器，据英国方面介绍，可以实现洲际远程打击。', + '那么这款无人机到底有多厉害？'] +from fastNLP.api import CWS +cws = CWS(device='cpu') +print(cws.predict(text)) +# ['编者 按 : 7月 12日 , 英国 航空 航天 系统 公司 公布 了 该 公司 研制 的 第一 款 高 科技 隐形 无人 机雷电 之 神 。', '这 款 飞行 从 外型 上 来 看 酷似 电影 中 的 太空 飞行器 , 据 英国 方面 介绍 , 可以 实现 洲际 远程 打击 。', '那么 这 款 无人 机 到底 有 多 厉害 ?'] +``` + +### 词性标注 +```python +# 输入已分词序列 +text = [['编者', '按：', '7月', '12日', '，', '英国', '航空', '航天', '系统', '公司', '公布', '了', '该', '公司', + '研制', '的', '第一款', '高科技', '隐形', '无人机', '雷电之神', '。'], + ['那么', '这', '款', '无人机', '到底', '有', '多', '厉害', '？']] +from fastNLP.api import POS +pos = POS(device='cpu') +print(pos.predict(text)) +# [['编者/NN', '按：/NN', '7月/NT', '12日/NT', '，/PU', '英国/NR', '航空/NN', '航天/NN', '系统/NN', '公司/NN', '公布/VV', '了/AS', '该/DT', '公司/NN', '研制/VV', '的/DEC', '第一款/NN', '高科技/NN', '隐形/AD', '无人机/VV', '雷电之神/NN', '。/PU'], ['那么/AD', '这/DT', '款/NN', '无人机/VV', '到底/AD', '有/VE', '多/AD', '厉害/VA', '？/PU']] +``` + +### 句法分析 +```python +text = [['编者', '按：', '7月', '12日', '，', '英国', '航空', '航天', '系统', '公司', '公布', '了', '该', '公司', + '研制', '的', '第一款', '高科技', '隐形', '无人机', '雷电之神', '。'], + ['那么', '这', '款', '无人机', '到底', '有', '多', '厉害', '？']] +from fastNLP.api import Parser +parser = Parser(device='cpu') +print(parser.predict(text)) +# [['2/nn', '4/nn', '4/nn', '20/tmod', '11/punct', '10/nn', '10/nn', '10/nn', '10/nn', '11/nsubj', '20/dep', '11/asp', '14/det', '15/nsubj', '18/rcmod', '15/cpm', '18/nn', '11/dobj', '20/advmod', '0/root', '20/dobj', '20/punct'], ['4/advmod', '3/det', '8/xsubj', '8/dep', '8/advmod', '8/dep', '8/advmod', '0/root', '8/punct']] +``` + +完整样例见`examples.py` \ No newline at end of file diff --git a/legacy/api/__init__.py b/legacy/api/__init__.py new file mode 100644 index 00000000..5171d8c2 --- /dev/null +++ b/legacy/api/__init__.py @@ -0,0 +1,2 @@ +__all__ = ["CWS", "POS", "Parser"] +from .api import CWS, POS, Parser diff --git a/fastNLP/api/api.py b/legacy/api/api.py similarity index 56% rename from fastNLP/api/api.py rename to legacy/api/api.py index 8368dcc9..d5d1df6b 100644 --- a/fastNLP/api/api.py +++ b/legacy/api/api.py @@ -6,33 +6,122 @@ warnings.filterwarnings('ignore') import os from fastNLP.core.dataset import DataSet - -from fastNLP.api.utils import load_url -from fastNLP.api.processor import ModelProcessor -from reproduction.chinese_word_segment.cws_io.cws_reader import ConllCWSReader -from reproduction.pos_tag_model.pos_reader import ZhConllPOSReader -from reproduction.Biaffine_parser.util import ConllxDataLoader, add_seg_tag +from .utils import load_url +from .processor import ModelProcessor +from fastNLP.io.dataset_loader import _cut_long_sentence, ConllLoader from fastNLP.core.instance import Instance -from fastNLP.api.pipeline import Pipeline +from ..api.pipeline import Pipeline from fastNLP.core.metrics import SpanFPreRecMetric -from fastNLP.api.processor import IndexerProcessor +from .processor import IndexerProcessor # TODO add pretrain urls model_urls = { - "cws": "http://123.206.98.91:8888/download/cws_crf_1_11-457fc899.pkl", - "pos": "http://123.206.98.91:8888/download/pos_tag_model_20190108-f3c60ee5.pkl", - "parser": "http://123.206.98.91:8888/download/biaffine_parser-3a2f052c.pkl" + "cws": "http://123.206.98.91:8888/download/cws_lstm_ctb9_1_20-09908656.pkl", + "pos": "http://123.206.98.91:8888/download/pos_tag_model_20190119-43f8b435.pkl", + "parser": "http://123.206.98.91:8888/download/parser_20190204-c72ca5c0.pkl" } +class ConllCWSReader(object): + """Deprecated. Use ConllLoader for all types of conll-format files.""" + + def __init__(self): + pass + + def load(self, path, cut_long_sent=False): + """ + 返回的DataSet只包含raw_sentence这个field，内容为str。 + 假定了输入为conll的格式，以空行隔开两个句子，每行共7列，即 + :: + + 1 编者按 编者按 NN O 11 nmod:topic + 2 ： ： PU O 11 punct + 3 7月 7月 NT DATE 4 compound:nn + 4 12日 12日 NT DATE 11 nmod:tmod + 5 ， ， PU O 11 punct + + 1 这 这 DT O 3 det + 2 款 款 M O 1 mark:clf + 3 飞行 飞行 NN O 8 nsubj + 4 从 从 P O 5 case + 5 外型 外型 NN O 8 nmod:prep + + """ + datalist = [] + with open(path, 'r', encoding='utf-8') as f: + sample = [] + for line in f: + if line.startswith('\n'): + datalist.append(sample) + sample = [] + elif line.startswith('#'): + continue + else: + sample.append(line.strip().split()) + if len(sample) > 0: + datalist.append(sample) + + ds = DataSet() + for sample in datalist: + # print(sample) + res = self.get_char_lst(sample) + if res is None: + continue + line = ' '.join(res) + if cut_long_sent: + sents = _cut_long_sentence(line) + else: + sents = [line] + for raw_sentence in sents: + ds.append(Instance(raw_sentence=raw_sentence)) + return ds + + def get_char_lst(self, sample): + if len(sample) == 0: + return None + text = [] + for w in sample: + t1, t2, t3, t4 = w[1], w[3], w[6], w[7] + if t3 == '_': + return None + text.append(t1) + return text + + +class ConllxDataLoader(ConllLoader): + """返回“词级别”的标签信息，包括词、词性、（句法）头依赖、（句法）边标签。跟``ZhConllPOSReader``完全不同。 + + Deprecated. Use ConllLoader for all types of conll-format files. + """ + + def __init__(self): + headers = [ + 'words', 'pos_tags', 'heads', 'labels', + ] + indexs = [ + 1, 3, 6, 7, + ] + super(ConllxDataLoader, self).__init__(headers=headers, indexes=indexs) + + class API: def __init__(self): self.pipeline = None self._dict = None - + def predict(self, *args, **kwargs): + """Do prediction for the given input. + """ raise NotImplementedError + + def test(self, file_path): + """Test performance over the given data set. + :param str file_path: + :return: a dictionary of metric values + """ + raise NotImplementedError + def load(self, path, device): if os.path.exists(os.path.expanduser(path)): _dict = torch.load(path, map_location='cpu') @@ -52,104 +141,98 @@ class POS(API): :param str device: device name such as "cpu" or "cuda:0". Use the same notation as PyTorch. """ - + def __init__(self, model_path=None, device='cpu'): super(POS, self).__init__() if model_path is None: model_path = model_urls['pos'] - + self.load(model_path, device) - + def predict(self, content): - """ - + """predict函数的介绍， + 函数介绍的第二句，这句话不会换行 + :param content: list of list of str. Each string is a token(word). :return answer: list of list of str. Each string is a tag. """ if not hasattr(self, "pipeline"): raise ValueError("You have to load model first.") - - sentence_list = [] + + sentence_list = content # 1. 检查sentence的类型 - if isinstance(content, str): - sentence_list.append(content) - elif isinstance(content, list): - sentence_list = content - + for sentence in sentence_list: + if not all((type(obj) == str for obj in sentence)): + raise ValueError("Input must be list of list of string.") + # 2. 组建dataset dataset = DataSet() dataset.add_field("words", sentence_list) - + # 3. 使用pipeline self.pipeline(dataset) - - def decode_tags(ins): - pred_tags = ins["tag"] - chars = ins["words"] - words = [] - start_idx = 0 - for idx, tag in enumerate(pred_tags): - if tag[0] == "S": - words.append(chars[start_idx:idx + 1] + "/" + tag[2:]) - start_idx = idx + 1 - elif tag[0] == "E": - words.append("".join(chars[start_idx:idx + 1]) + "/" + tag[2:]) - start_idx = idx + 1 - return words - - dataset.apply(decode_tags, new_field_name="tag_output") - - output = dataset.field_arrays["tag_output"].content + + def merge_tag(words_list, tags_list): + rtn = [] + for words, tags in zip(words_list, tags_list): + rtn.append([w + "/" + t for w, t in zip(words, tags)]) + return rtn + + output = dataset.field_arrays["tag"].content if isinstance(content, str): return output[0] elif isinstance(content, list): - return output - + return merge_tag(content, output) + def test(self, file_path): - test_data = ZhConllPOSReader().load(file_path) - - tag_vocab = self._dict["tag_vocab"] - pipeline = self._dict["pipeline"] + test_data = ConllxDataLoader().load(file_path) + + save_dict = self._dict + tag_vocab = save_dict["tag_vocab"] + pipeline = save_dict["pipeline"] index_tag = IndexerProcessor(vocab=tag_vocab, field_name="tag", new_added_field_name="truth", is_input=False) pipeline.pipeline = [index_tag] + pipeline.pipeline - + + test_data.rename_field("pos_tags", "tag") pipeline(test_data) test_data.set_target("truth") prediction = test_data.field_arrays["predict"].content truth = test_data.field_arrays["truth"].content seq_len = test_data.field_arrays["word_seq_origin_len"].content - + # padding by hand max_length = max([len(seq) for seq in prediction]) for idx in range(len(prediction)): prediction[idx] = list(prediction[idx]) + ([0] * (max_length - len(prediction[idx]))) truth[idx] = list(truth[idx]) + ([0] * (max_length - len(truth[idx]))) evaluator = SpanFPreRecMetric(tag_vocab=tag_vocab, pred="predict", target="truth", - seq_lens="word_seq_origin_len") + seq_len="word_seq_origin_len") evaluator({"predict": torch.Tensor(prediction), "word_seq_origin_len": torch.Tensor(seq_len)}, {"truth": torch.Tensor(truth)}) test_result = evaluator.get_metric() f1 = round(test_result['f'] * 100, 2) pre = round(test_result['pre'] * 100, 2) rec = round(test_result['rec'] * 100, 2) - + return {"F1": f1, "precision": pre, "recall": rec} class CWS(API): - def __init__(self, model_path=None, device='cpu'): - """ - 中文分词高级接口。 + """ + 中文分词高级接口。 - :param model_path: 当model_path为None，使用默认位置的model。如果默认位置不存在，则自动下载模型 - :param device: str，可以为'cpu', 'cuda'或'cuda:0'等。会将模型load到相应device进行推断。 - """ + :param model_path: 当model_path为None，使用默认位置的model。如果默认位置不存在，则自动下载模型 + :param device: str，可以为'cpu', 'cuda'或'cuda:0'等。会将模型load到相应device进行推断。 + """ + + def __init__(self, model_path=None, device='cpu'): + super(CWS, self).__init__() if model_path is None: model_path = model_urls['cws'] - + self.load(model_path, device) - + def predict(self, content): """ 分词接口。 @@ -160,42 +243,44 @@ class CWS(API): """ if not hasattr(self, 'pipeline'): raise ValueError("You have to load model first.") - + sentence_list = [] # 1. 检查sentence的类型 if isinstance(content, str): sentence_list.append(content) elif isinstance(content, list): sentence_list = content - + # 2. 组建dataset dataset = DataSet() dataset.add_field('raw_sentence', sentence_list) - + # 3. 使用pipeline self.pipeline(dataset) - + output = dataset.get_field('output').content if isinstance(content, str): return output[0] elif isinstance(content, list): return output - + def test(self, filepath): """ 传入一个分词文件路径，返回该数据集上分词f1, precision, recall。 - 分词文件应该为: + 分词文件应该为:: + 1 编者按 编者按 NN O 11 nmod:topic 2 ： ： PU O 11 punct 3 7月 7月 NT DATE 4 compound:nn 4 12日 12日 NT DATE 11 nmod:tmod 5 ， ， PU O 11 punct - + 1 这 这 DT O 3 det 2 款 款 M O 1 mark:clf 3 飞行 飞行 NN O 8 nsubj 4 从 从 P O 5 case 5 外型 外型 NN O 8 nmod:prep + 以空行分割两个句子，有内容的每行有7列。 :param filepath: str, 文件路径路径。 @@ -204,29 +289,29 @@ class CWS(API): tag_proc = self._dict['tag_proc'] cws_model = self.pipeline.pipeline[-2].model pipeline = self.pipeline.pipeline[:-2] - + pipeline.insert(1, tag_proc) pp = Pipeline(pipeline) - + reader = ConllCWSReader() - + # te_filename = '/home/hyan/ctb3/test.conllx' te_dataset = reader.load(filepath) pp(te_dataset) - - from fastNLP.core.tester import Tester - from fastNLP.core.metrics import BMESF1PreRecMetric - - tester = Tester(data=te_dataset, model=cws_model, metrics=BMESF1PreRecMetric(target='target'), batch_size=64, + + from ..core.tester import Tester + from ..core.metrics import SpanFPreRecMetric + + tester = Tester(data=te_dataset, model=cws_model, metrics=SpanFPreRecMetric(tag_proc.get_vocab()), batch_size=64, verbose=0) eval_res = tester.test() - - f1 = eval_res['BMESF1PreRecMetric']['f'] - pre = eval_res['BMESF1PreRecMetric']['pre'] - rec = eval_res['BMESF1PreRecMetric']['rec'] + + f1 = eval_res['SpanFPreRecMetric']['f'] + pre = eval_res['SpanFPreRecMetric']['pre'] + rec = eval_res['SpanFPreRecMetric']['rec'] # print("f1:{:.2f}, pre:{:.2f}, rec:{:.2f}".format(f1, pre, rec)) - - return f1, pre, rec + + return {"F1": f1, "precision": pre, "recall": rec} class Parser(API): @@ -234,24 +319,25 @@ class Parser(API): super(Parser, self).__init__() if model_path is None: model_path = model_urls['parser'] - + self.pos_tagger = POS(device=device) self.load(model_path, device) - + def predict(self, content): if not hasattr(self, 'pipeline'): raise ValueError("You have to load model first.") - + # 1. 利用POS得到分词和pos tagging结果 pos_out = self.pos_tagger.predict(content) # pos_out = ['这里/NN 是/VB 分词/NN 结果/NN'.split()] - + # 2. 组建dataset dataset = DataSet() dataset.add_field('wp', pos_out) dataset.apply(lambda x: [''] + [w.split('/')[0] for w in x['wp']], new_field_name='words') dataset.apply(lambda x: [''] + [w.split('/')[1] for w in x['wp']], new_field_name='pos') - + dataset.rename_field("words", "raw_words") + # 3. 使用pipeline self.pipeline(dataset) dataset.apply(lambda x: [str(arc) for arc in x['arc_pred']], new_field_name='arc_pred') @@ -259,49 +345,92 @@ class Parser(API): zip(x['arc_pred'], x['label_pred_seq'])][1:], new_field_name='output') # output like: [['2/top', '0/root', '4/nn', '2/dep']] return dataset.field_arrays['output'].content - + + def load_test_file(self, path): + def get_one(sample): + sample = list(map(list, zip(*sample))) + if len(sample) == 0: + return None + for w in sample[7]: + if w == '_': + print('Error Sample {}'.format(sample)) + return None + # return word_seq, pos_seq, head_seq, head_tag_seq + return sample[1], sample[3], list(map(int, sample[6])), sample[7] + + datalist = [] + with open(path, 'r', encoding='utf-8') as f: + sample = [] + for line in f: + if line.startswith('\n'): + datalist.append(sample) + sample = [] + elif line.startswith('#'): + continue + else: + sample.append(line.split('\t')) + if len(sample) > 0: + datalist.append(sample) + + data = [get_one(sample) for sample in datalist] + data_list = list(filter(lambda x: x is not None, data)) + return data_list + def test(self, filepath): - data = ConllxDataLoader().load(filepath) - ds = DataSet() - for ins1, ins2 in zip(add_seg_tag(data), data): - ds.append(Instance(words=ins1[0], tag=ins1[1], - gold_words=ins2[0], gold_pos=ins2[1], - gold_heads=ins2[2], gold_head_tags=ins2[3])) - + data = self.load_test_file(filepath) + + def convert(data): + BOS = '' + dataset = DataSet() + for sample in data: + word_seq = [BOS] + sample[0] + pos_seq = [BOS] + sample[1] + heads = [0] + sample[2] + head_tags = [BOS] + sample[3] + dataset.append(Instance(raw_words=word_seq, + pos=pos_seq, + gold_heads=heads, + arc_true=heads, + tags=head_tags)) + return dataset + + ds = convert(data) pp = self.pipeline for p in pp: if p.field_name == 'word_list': p.field_name = 'gold_words' elif p.field_name == 'pos_list': p.field_name = 'gold_pos' + # ds.rename_field("words", "raw_words") + # ds.rename_field("tag", "pos") pp(ds) head_cor, label_cor, total = 0, 0, 0 for ins in ds: head_gold = ins['gold_heads'] - head_pred = ins['heads'] + head_pred = ins['arc_pred'] length = len(head_gold) total += length for i in range(length): head_cor += 1 if head_pred[i] == head_gold[i] else 0 uas = head_cor / total - print('uas:{:.2f}'.format(uas)) - + # print('uas:{:.2f}'.format(uas)) + for p in pp: if p.field_name == 'gold_words': p.field_name = 'word_list' elif p.field_name == 'gold_pos': p.field_name = 'pos_list' - - return uas + + return {"USA": round(uas, 5)} class Analyzer: def __init__(self, device='cpu'): - + self.cws = CWS(device=device) self.pos = POS(device=device) self.parser = Parser(device=device) - + def predict(self, content, seg=False, pos=False, parser=False): if seg is False and pos is False and parser is False: seg = True @@ -315,9 +444,9 @@ class Analyzer: if parser: parser_output = self.parser.predict(content) output_dict['parser'] = parser_output - + return output_dict - + def test(self, filepath): output_dict = {} if self.cws: @@ -329,5 +458,5 @@ class Analyzer: if self.parser: parser_output = self.parser.test(filepath) output_dict['parser'] = parser_output - + return output_dict diff --git a/fastNLP/api/converter.py b/legacy/api/converter.py similarity index 100% rename from fastNLP/api/converter.py rename to legacy/api/converter.py diff --git a/legacy/api/examples.py b/legacy/api/examples.py new file mode 100644 index 00000000..c1b2e155 --- /dev/null +++ b/legacy/api/examples.py @@ -0,0 +1,56 @@ +""" +api/example.py contains all API examples provided by fastNLP. +It is used as a tutorial for API or a test script since it is difficult to test APIs in travis. + +""" +from . import CWS, POS, Parser + +text = ['编者按：7月12日，英国航空航天系统公司公布了该公司研制的第一款高科技隐形无人机雷电之神。', + '这款飞行从外型上来看酷似电影中的太空飞行器，据英国方面介绍，可以实现洲际远程打击。', + '那么这款无人机到底有多厉害？'] + + +def chinese_word_segmentation(): + cws = CWS(device='cpu') + print(cws.predict(text)) + + +def chinese_word_segmentation_test(): + cws = CWS(device='cpu') + print(cws.test("../../test/data_for_tests/zh_sample.conllx")) + + +def pos_tagging(): + # 输入已分词序列 + text = [['编者', '按：', '7月', '12日', '，', '英国', '航空', '航天', '系统', '公司', '公布', '了', '该', '公司', + '研制', '的', '第一款', '高科技', '隐形', '无人机', '雷电之神', '。'], + ['那么', '这', '款', '无人机', '到底', '有', '多', '厉害', '？']] + pos = POS(device='cpu') + print(pos.predict(text)) + + +def pos_tagging_test(): + pos = POS(device='cpu') + print(pos.test("../../test/data_for_tests/zh_sample.conllx")) + + +def syntactic_parsing(): + text = [['编者', '按：', '7月', '12日', '，', '英国', '航空', '航天', '系统', '公司', '公布', '了', '该', '公司', + '研制', '的', '第一款', '高科技', '隐形', '无人机', '雷电之神', '。'], + ['那么', '这', '款', '无人机', '到底', '有', '多', '厉害', '？']] + parser = Parser(device='cpu') + print(parser.predict(text)) + + +def syntactic_parsing_test(): + parser = Parser(device='cpu') + print(parser.test("../../test/data_for_tests/zh_sample.conllx")) + + +if __name__ == "__main__": + # chinese_word_segmentation() + # chinese_word_segmentation_test() + # pos_tagging() + # pos_tagging_test() + syntactic_parsing() + # syntactic_parsing_test() diff --git a/fastNLP/api/pipeline.py b/legacy/api/pipeline.py similarity index 95% rename from fastNLP/api/pipeline.py rename to legacy/api/pipeline.py index 0c567678..2cec16b3 100644 --- a/fastNLP/api/pipeline.py +++ b/legacy/api/pipeline.py @@ -1,4 +1,4 @@ -from fastNLP.api.processor import Processor +from ..api.processor import Processor class Pipeline: diff --git a/fastNLP/api/processor.py b/legacy/api/processor.py similarity index 73% rename from fastNLP/api/processor.py rename to legacy/api/processor.py index 7354fe0f..4c442ed2 100644 --- a/fastNLP/api/processor.py +++ b/legacy/api/processor.py @@ -102,6 +102,7 @@ class PreAppendProcessor(Processor): [data] + instance[field_name] """ + def __init__(self, data, field_name, new_added_field_name=None): super(PreAppendProcessor, self).__init__(field_name, new_added_field_name) self.data = data @@ -116,6 +117,7 @@ class SliceProcessor(Processor): 从某个field中只取部分内容。等价于instance[field_name][start:end:step] """ + def __init__(self, start, end, step, field_name, new_added_field_name=None): super(SliceProcessor, self).__init__(field_name, new_added_field_name) for o in (start, end, step): @@ -132,6 +134,7 @@ class Num2TagProcessor(Processor): 将一句话中的数字转换为某个tag。 """ + def __init__(self, tag, field_name, new_added_field_name=None): """ @@ -163,6 +166,7 @@ class IndexerProcessor(Processor): 给定一个vocabulary , 将指定field转换为index形式。指定field应该是一维的list，比如 ['我', '是', xxx] """ + def __init__(self, vocab, field_name, new_added_field_name, delete_old_field=False, is_input=True): assert isinstance(vocab, Vocabulary), "Only Vocabulary class is allowed, not {}.".format(type(vocab)) @@ -215,6 +219,7 @@ class SeqLenProcessor(Processor): 根据某个field新增一个sequence length的field。取该field的第一维 """ + def __init__(self, field_name, new_added_field_name='seq_lens', is_input=True): super(SeqLenProcessor, self).__init__(field_name, new_added_field_name) self.is_input = is_input @@ -229,6 +234,7 @@ class SeqLenProcessor(Processor): from fastNLP.core.utils import _build_args + class ModelProcessor(Processor): def __init__(self, model, seq_len_field_name='seq_lens', batch_size=32): """ @@ -251,10 +257,7 @@ class ModelProcessor(Processor): data_iterator = Batch(dataset, batch_size=self.batch_size, sampler=SequentialSampler()) batch_output = defaultdict(list) - if hasattr(self.model, "predict"): - predict_func = self.model.predict - else: - predict_func = self.model.forward + predict_func = self.model.forward with torch.no_grad(): for batch_x, _ in data_iterator: refined_batch_x = _build_args(predict_func, **batch_x) @@ -292,6 +295,7 @@ class Index2WordProcessor(Processor): 将DataSet中某个为index的field根据vocab转换为str """ + def __init__(self, vocab, field_name, new_added_field_name): super(Index2WordProcessor, self).__init__(field_name, new_added_field_name) self.vocab = vocab @@ -303,7 +307,6 @@ class Index2WordProcessor(Processor): class SetTargetProcessor(Processor): - # TODO; remove it. def __init__(self, *fields, flag=True): super(SetTargetProcessor, self).__init__(None, None) self.fields = fields @@ -313,6 +316,7 @@ class SetTargetProcessor(Processor): dataset.set_target(*self.fields, flag=self.flag) return dataset + class SetInputProcessor(Processor): def __init__(self, *fields, flag=True): super(SetInputProcessor, self).__init__(None, None) @@ -322,3 +326,103 @@ class SetInputProcessor(Processor): def process(self, dataset): dataset.set_input(*self.fields, flag=self.flag) return dataset + + +class VocabIndexerProcessor(Processor): + """ + 根据DataSet创建Vocabulary，并将其用数字index。新生成的index的field会被放在new_added_filed_name, 如果没有提供 + new_added_field_name, 则覆盖原有的field_name. + + """ + + def __init__(self, field_name, new_added_filed_name=None, min_freq=1, max_size=None, + verbose=0, is_input=True): + """ + + :param field_name: 从哪个field_name创建词表，以及对哪个field_name进行index操作 + :param new_added_filed_name: index时，生成的index field的名称，如果不传入，则覆盖field_name. + :param min_freq: 创建的Vocabulary允许的单词最少出现次数. + :param max_size: 创建的Vocabulary允许的最大的单词数量 + :param verbose: 0, 不输出任何信息；1，输出信息 + :param bool is_input: + """ + super(VocabIndexerProcessor, self).__init__(field_name, new_added_filed_name) + self.min_freq = min_freq + self.max_size = max_size + + self.verbose = verbose + self.is_input = is_input + + def construct_vocab(self, *datasets): + """ + 使用传入的DataSet创建vocabulary + + :param datasets: DataSet类型的数据，用于构建vocabulary + :return: + """ + self.vocab = Vocabulary(min_freq=self.min_freq, max_size=self.max_size) + for dataset in datasets: + assert isinstance(dataset, DataSet), "Only Dataset class is allowed, not {}.".format(type(dataset)) + dataset.apply(lambda ins: self.vocab.update(ins[self.field_name])) + self.vocab.build_vocab() + if self.verbose: + print("Vocabulary Constructed, has {} items.".format(len(self.vocab))) + + def process(self, *datasets, only_index_dataset=None): + """ + 若还未建立Vocabulary，则使用dataset中的DataSet建立vocabulary；若已经有了vocabulary则使用已有的vocabulary。得到vocabulary + 后，则会index datasets与only_index_dataset。 + + :param datasets: DataSet类型的数据 + :param only_index_dataset: DataSet, or list of DataSet. 该参数中的内容只会被用于index，不会被用于生成vocabulary。 + :return: + """ + if len(datasets) == 0 and not hasattr(self, 'vocab'): + raise RuntimeError("You have to construct vocabulary first. Or you have to pass datasets to construct it.") + if not hasattr(self, 'vocab'): + self.construct_vocab(*datasets) + else: + if self.verbose: + print("Using constructed vocabulary with {} items.".format(len(self.vocab))) + to_index_datasets = [] + if len(datasets) != 0: + for dataset in datasets: + assert isinstance(dataset, DataSet), "Only DataSet class is allowed, not {}.".format(type(dataset)) + to_index_datasets.append(dataset) + + if not (only_index_dataset is None): + if isinstance(only_index_dataset, list): + for dataset in only_index_dataset: + assert isinstance(dataset, DataSet), "Only DataSet class is allowed, not {}.".format(type(dataset)) + to_index_datasets.append(dataset) + elif isinstance(only_index_dataset, DataSet): + to_index_datasets.append(only_index_dataset) + else: + raise TypeError('Only DataSet or list of DataSet is allowed, not {}.'.format(type(only_index_dataset))) + + for dataset in to_index_datasets: + assert isinstance(dataset, DataSet), "Only DataSet class is allowed, not {}.".format(type(dataset)) + dataset.apply(lambda ins: [self.vocab.to_index(token) for token in ins[self.field_name]], + new_field_name=self.new_added_field_name, is_input=self.is_input) + # 只返回一个，infer时为了跟其他processor保持一致 + if len(to_index_datasets) == 1: + return to_index_datasets[0] + + def set_vocab(self, vocab): + assert isinstance(vocab, Vocabulary), "Only fastNLP.core.Vocabulary is allowed, not {}.".format(type(vocab)) + self.vocab = vocab + + def delete_vocab(self): + del self.vocab + + def get_vocab_size(self): + return len(self.vocab) + + def set_verbose(self, verbose): + """ + 设置processor verbose状态。 + + :param verbose: int, 0，不输出任何信息；1，输出vocab 信息。 + :return: + """ + self.verbose = verbose diff --git a/fastNLP/api/utils.py b/legacy/api/utils.py similarity index 97% rename from fastNLP/api/utils.py rename to legacy/api/utils.py index a54a53d9..184e5fe6 100644 --- a/fastNLP/api/utils.py +++ b/legacy/api/utils.py @@ -20,10 +20,10 @@ except ImportError: from urllib.request import urlopen from urllib.parse import urlparse try: - from tqdm import tqdm -except ImportError: - tqdm = None # defined below - + from tqdm.auto import tqdm +except: + from fastNLP.core.utils import _pseudo_tqdm as tqdm + # matches bfd8deac from resnet18-bfd8deac.pth HASH_REGEX = re.compile(r'-([a-f0-9]*)\.') diff --git a/reproduction/chinese_word_segment/cws_io/__init__.py b/legacy/automl/__init__.py similarity index 100% rename from reproduction/chinese_word_segment/cws_io/__init__.py rename to legacy/automl/__init__.py diff --git a/legacy/automl/enas_controller.py b/legacy/automl/enas_controller.py new file mode 100644 index 00000000..6ddbb211 --- /dev/null +++ b/legacy/automl/enas_controller.py @@ -0,0 +1,223 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch +"""A module with NAS controller-related code.""" +import collections +import os + +import torch +import torch.nn.functional as F + +import fastNLP.automl.enas_utils as utils +from fastNLP.automl.enas_utils import Node + + +def _construct_dags(prev_nodes, activations, func_names, num_blocks): + """Constructs a set of DAGs based on the actions, i.e., previous nodes and + activation functions, sampled from the controller/policy pi. + + Args: + prev_nodes: Previous node actions from the policy. + activations: Activations sampled from the policy. + func_names: Mapping from activation function names to functions. + num_blocks: Number of blocks in the target RNN cell. + + Returns: + A list of DAGs defined by the inputs. + + RNN cell DAGs are represented in the following way: + + 1. Each element (node) in a DAG is a list of `Node`s. + + 2. The `Node`s in the list dag[i] correspond to the subsequent nodes + that take the output from node i as their own input. + + 3. dag[-1] is the node that takes input from x^{(t)} and h^{(t - 1)}. + dag[-1] always feeds dag[0]. + dag[-1] acts as if `w_xc`, `w_hc`, `w_xh` and `w_hh` are its + weights. + + 4. dag[N - 1] is the node that produces the hidden state passed to + the next timestep. dag[N - 1] is also always a leaf node, and therefore + is always averaged with the other leaf nodes and fed to the output + decoder. + """ + dags = [] + for nodes, func_ids in zip(prev_nodes, activations): + dag = collections.defaultdict(list) + + # add first node + dag[-1] = [Node(0, func_names[func_ids[0]])] + dag[-2] = [Node(0, func_names[func_ids[0]])] + + # add following nodes + for jdx, (idx, func_id) in enumerate(zip(nodes, func_ids[1:])): + dag[utils.to_item(idx)].append(Node(jdx + 1, func_names[func_id])) + + leaf_nodes = set(range(num_blocks)) - dag.keys() + + # merge with avg + for idx in leaf_nodes: + dag[idx] = [Node(num_blocks, 'avg')] + + # This is actually y^{(t)}. h^{(t)} is node N - 1 in + # the graph, where N Is the number of nodes. I.e., h^{(t)} takes + # only one other node as its input. + # last h[t] node + last_node = Node(num_blocks + 1, 'h[t]') + dag[num_blocks] = [last_node] + dags.append(dag) + + return dags + + +class Controller(torch.nn.Module): + """Based on + https://github.com/pytorch/examples/blob/master/word_language_model/model.py + + RL controllers do not necessarily have much to do with + language models. + + Base the controller RNN on the GRU from: + https://github.com/ikostrikov/pytorch-a2c-ppo-acktr/blob/master/model.py + """ + def __init__(self, num_blocks=4, controller_hid=100, cuda=False): + torch.nn.Module.__init__(self) + + # `num_tokens` here is just the activation function + # for every even step, + self.shared_rnn_activations = ['tanh', 'ReLU', 'identity', 'sigmoid'] + self.num_tokens = [len(self.shared_rnn_activations)] + self.controller_hid = controller_hid + self.use_cuda = cuda + self.num_blocks = num_blocks + for idx in range(num_blocks): + self.num_tokens += [idx + 1, len(self.shared_rnn_activations)] + self.func_names = self.shared_rnn_activations + + num_total_tokens = sum(self.num_tokens) + + self.encoder = torch.nn.Embedding(num_total_tokens, + controller_hid) + self.lstm = torch.nn.LSTMCell(controller_hid, controller_hid) + + # Perhaps these weights in the decoder should be + # shared? At least for the activation functions, which all have the + # same size. + self.decoders = [] + for idx, size in enumerate(self.num_tokens): + decoder = torch.nn.Linear(controller_hid, size) + self.decoders.append(decoder) + + self._decoders = torch.nn.ModuleList(self.decoders) + + self.reset_parameters() + self.static_init_hidden = utils.keydefaultdict(self.init_hidden) + + def _get_default_hidden(key): + return utils.get_variable( + torch.zeros(key, self.controller_hid), + self.use_cuda, + requires_grad=False) + + self.static_inputs = utils.keydefaultdict(_get_default_hidden) + + def reset_parameters(self): + init_range = 0.1 + for param in self.parameters(): + param.data.uniform_(-init_range, init_range) + for decoder in self.decoders: + decoder.bias.data.fill_(0) + + def forward(self, # pylint:disable=arguments-differ + inputs, + hidden, + block_idx, + is_embed): + if not is_embed: + embed = self.encoder(inputs) + else: + embed = inputs + + hx, cx = self.lstm(embed, hidden) + logits = self.decoders[block_idx](hx) + + logits /= 5.0 + + # # exploration + # if self.args.mode == 'train': + # logits = (2.5 * F.tanh(logits)) + + return logits, (hx, cx) + + def sample(self, batch_size=1, with_details=False, save_dir=None): + """Samples a set of `args.num_blocks` many computational nodes from the + controller, where each node is made up of an activation function, and + each node except the last also includes a previous node. + """ + if batch_size < 1: + raise Exception(f'Wrong batch_size: {batch_size} < 1') + + # [B, L, H] + inputs = self.static_inputs[batch_size] + hidden = self.static_init_hidden[batch_size] + + activations = [] + entropies = [] + log_probs = [] + prev_nodes = [] + # The RNN controller alternately outputs an activation, + # followed by a previous node, for each block except the last one, + # which only gets an activation function. The last node is the output + # node, and its previous node is the average of all leaf nodes. + for block_idx in range(2*(self.num_blocks - 1) + 1): + logits, hidden = self.forward(inputs, + hidden, + block_idx, + is_embed=(block_idx == 0)) + + probs = F.softmax(logits, dim=-1) + log_prob = F.log_softmax(logits, dim=-1) + # .mean() for entropy? + entropy = -(log_prob * probs).sum(1, keepdim=False) + + action = probs.multinomial(num_samples=1).data + selected_log_prob = log_prob.gather( + 1, utils.get_variable(action, requires_grad=False)) + + # why the [:, 0] here? Should it be .squeeze(), or + # .view()? Same below with `action`. + entropies.append(entropy) + log_probs.append(selected_log_prob[:, 0]) + + # 0: function, 1: previous node + mode = block_idx % 2 + inputs = utils.get_variable( + action[:, 0] + sum(self.num_tokens[:mode]), + requires_grad=False) + + if mode == 0: + activations.append(action[:, 0]) + elif mode == 1: + prev_nodes.append(action[:, 0]) + + prev_nodes = torch.stack(prev_nodes).transpose(0, 1) + activations = torch.stack(activations).transpose(0, 1) + + dags = _construct_dags(prev_nodes, + activations, + self.func_names, + self.num_blocks) + + if save_dir is not None: + for idx, dag in enumerate(dags): + utils.draw_network(dag, + os.path.join(save_dir, f'graph{idx}.png')) + + if with_details: + return dags, torch.cat(log_probs), torch.cat(entropies) + + return dags + + def init_hidden(self, batch_size): + zeros = torch.zeros(batch_size, self.controller_hid) + return (utils.get_variable(zeros, self.use_cuda, requires_grad=False), + utils.get_variable(zeros.clone(), self.use_cuda, requires_grad=False)) diff --git a/legacy/automl/enas_model.py b/legacy/automl/enas_model.py new file mode 100644 index 00000000..4f9fb449 --- /dev/null +++ b/legacy/automl/enas_model.py @@ -0,0 +1,388 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch + +"""Module containing the shared RNN model.""" +import collections + +import numpy as np +import torch +import torch.nn.functional as F +from torch import nn +from torch.autograd import Variable + +import fastNLP.automl.enas_utils as utils +from fastNLP.models.base_model import BaseModel + + +def _get_dropped_weights(w_raw, dropout_p, is_training): + """Drops out weights to implement DropConnect. + + Args: + w_raw: Full, pre-dropout, weights to be dropped out. + dropout_p: Proportion of weights to drop out. + is_training: True iff _shared_ model is training. + + Returns: + The dropped weights. + + Why does torch.nn.functional.dropout() return: + 1. `torch.autograd.Variable()` on the training loop + 2. `torch.nn.Parameter()` on the controller or eval loop, when + training = False... + + Even though the call to `_setweights` in the Smerity repo's + `weight_drop.py` does not have this behaviour, and `F.dropout` always + returns `torch.autograd.Variable` there, even when `training=False`? + + The above TODO is the reason for the hacky check for `torch.nn.Parameter`. + """ + dropped_w = F.dropout(w_raw, p=dropout_p, training=is_training) + + if isinstance(dropped_w, torch.nn.Parameter): + dropped_w = dropped_w.clone() + + return dropped_w + +class EmbeddingDropout(torch.nn.Embedding): + """Class for dropping out embeddings by zero'ing out parameters in the + embedding matrix. + + This is equivalent to dropping out particular words, e.g., in the sentence + 'the quick brown fox jumps over the lazy dog', dropping out 'the' would + lead to the sentence '### quick brown fox jumps over ### lazy dog' (in the + embedding vector space). + + See 'A Theoretically Grounded Application of Dropout in Recurrent Neural + Networks', (Gal and Ghahramani, 2016). + """ + def __init__(self, + num_embeddings, + embedding_dim, + max_norm=None, + norm_type=2, + scale_grad_by_freq=False, + sparse=False, + dropout=0.1, + scale=None): + """Embedding constructor. + + Args: + dropout: Dropout probability. + scale: Used to scale parameters of embedding weight matrix that are + not dropped out. Note that this is _in addition_ to the + `1/(1 - dropout)` scaling. + + See `torch.nn.Embedding` for remaining arguments. + """ + torch.nn.Embedding.__init__(self, + num_embeddings=num_embeddings, + embedding_dim=embedding_dim, + max_norm=max_norm, + norm_type=norm_type, + scale_grad_by_freq=scale_grad_by_freq, + sparse=sparse) + self.dropout = dropout + assert (dropout >= 0.0) and (dropout < 1.0), ('Dropout must be >= 0.0 ' + 'and < 1.0') + self.scale = scale + + def forward(self, inputs): # pylint:disable=arguments-differ + """Embeds `inputs` with the dropped out embedding weight matrix.""" + if self.training: + dropout = self.dropout + else: + dropout = 0 + + if dropout: + mask = self.weight.data.new(self.weight.size(0), 1) + mask.bernoulli_(1 - dropout) + mask = mask.expand_as(self.weight) + mask = mask / (1 - dropout) + masked_weight = self.weight * Variable(mask) + else: + masked_weight = self.weight + if self.scale and self.scale != 1: + masked_weight = masked_weight * self.scale + + return F.embedding(inputs, + masked_weight, + max_norm=self.max_norm, + norm_type=self.norm_type, + scale_grad_by_freq=self.scale_grad_by_freq, + sparse=self.sparse) + + +class LockedDropout(nn.Module): + # code from https://github.com/salesforce/awd-lstm-lm/blob/master/locked_dropout.py + def __init__(self): + super().__init__() + + def forward(self, x, dropout=0.5): + if not self.training or not dropout: + return x + m = x.data.new(1, x.size(1), x.size(2)).bernoulli_(1 - dropout) + mask = Variable(m, requires_grad=False) / (1 - dropout) + mask = mask.expand_as(x) + return mask * x + + +class ENASModel(BaseModel): + """Shared RNN model.""" + def __init__(self, embed_num, num_classes, num_blocks=4, cuda=False, shared_hid=1000, shared_embed=1000): + super(ENASModel, self).__init__() + + self.use_cuda = cuda + + self.shared_hid = shared_hid + self.num_blocks = num_blocks + self.decoder = nn.Linear(self.shared_hid, num_classes) + self.encoder = EmbeddingDropout(embed_num, + shared_embed, + dropout=0.1) + self.lockdrop = LockedDropout() + self.dag = None + + # Tie weights + # self.decoder.weight = self.encoder.weight + + # Since W^{x, c} and W^{h, c} are always summed, there + # is no point duplicating their bias offset parameter. Likewise for + # W^{x, h} and W^{h, h}. + self.w_xc = nn.Linear(shared_embed, self.shared_hid) + self.w_xh = nn.Linear(shared_embed, self.shared_hid) + + # The raw weights are stored here because the hidden-to-hidden weights + # are weight dropped on the forward pass. + self.w_hc_raw = torch.nn.Parameter( + torch.Tensor(self.shared_hid, self.shared_hid)) + self.w_hh_raw = torch.nn.Parameter( + torch.Tensor(self.shared_hid, self.shared_hid)) + self.w_hc = None + self.w_hh = None + + self.w_h = collections.defaultdict(dict) + self.w_c = collections.defaultdict(dict) + + for idx in range(self.num_blocks): + for jdx in range(idx + 1, self.num_blocks): + self.w_h[idx][jdx] = nn.Linear(self.shared_hid, + self.shared_hid, + bias=False) + self.w_c[idx][jdx] = nn.Linear(self.shared_hid, + self.shared_hid, + bias=False) + + self._w_h = nn.ModuleList([self.w_h[idx][jdx] + for idx in self.w_h + for jdx in self.w_h[idx]]) + self._w_c = nn.ModuleList([self.w_c[idx][jdx] + for idx in self.w_c + for jdx in self.w_c[idx]]) + + self.batch_norm = None + # if args.mode == 'train': + # self.batch_norm = nn.BatchNorm1d(self.shared_hid) + # else: + # self.batch_norm = None + + self.reset_parameters() + self.static_init_hidden = utils.keydefaultdict(self.init_hidden) + + def setDAG(self, dag): + if self.dag is None: + self.dag = dag + + def forward(self, word_seq, hidden=None): + inputs = torch.transpose(word_seq, 0, 1) + + time_steps = inputs.size(0) + batch_size = inputs.size(1) + + + self.w_hh = _get_dropped_weights(self.w_hh_raw, + 0.5, + self.training) + self.w_hc = _get_dropped_weights(self.w_hc_raw, + 0.5, + self.training) + + # hidden = self.static_init_hidden[batch_size] if hidden is None else hidden + hidden = self.static_init_hidden[batch_size] + + embed = self.encoder(inputs) + + embed = self.lockdrop(embed, 0.65 if self.training else 0) + + # The norm of hidden states are clipped here because + # otherwise ENAS is especially prone to exploding activations on the + # forward pass. This could probably be fixed in a more elegant way, but + # it might be exposing a weakness in the ENAS algorithm as currently + # proposed. + # + # For more details, see + # https://github.com/carpedm20/ENAS-pytorch/issues/6 + clipped_num = 0 + max_clipped_norm = 0 + h1tohT = [] + logits = [] + for step in range(time_steps): + x_t = embed[step] + logit, hidden = self.cell(x_t, hidden, self.dag) + + hidden_norms = hidden.norm(dim=-1) + max_norm = 25.0 + if hidden_norms.data.max() > max_norm: + # Just directly use the torch slice operations + # in PyTorch v0.4. + # + # This workaround for PyTorch v0.3.1 does everything in numpy, + # because the PyTorch slicing and slice assignment is too + # flaky. + hidden_norms = hidden_norms.data.cpu().numpy() + + clipped_num += 1 + if hidden_norms.max() > max_clipped_norm: + max_clipped_norm = hidden_norms.max() + + clip_select = hidden_norms > max_norm + clip_norms = hidden_norms[clip_select] + + mask = np.ones(hidden.size()) + normalizer = max_norm/clip_norms + normalizer = normalizer[:, np.newaxis] + + mask[clip_select] = normalizer + + if self.use_cuda: + hidden *= torch.autograd.Variable( + torch.FloatTensor(mask).cuda(), requires_grad=False) + else: + hidden *= torch.autograd.Variable( + torch.FloatTensor(mask), requires_grad=False) + logits.append(logit) + h1tohT.append(hidden) + + h1tohT = torch.stack(h1tohT) + output = torch.stack(logits) + raw_output = output + + output = self.lockdrop(output, 0.4 if self.training else 0) + + #Pooling + output = torch.mean(output, 0) + + decoded = self.decoder(output) + + extra_out = {'dropped': decoded, + 'hiddens': h1tohT, + 'raw': raw_output} + return {'pred': decoded, 'hidden': hidden, 'extra_out': extra_out} + + def cell(self, x, h_prev, dag): + """Computes a single pass through the discovered RNN cell.""" + c = {} + h = {} + f = {} + + f[0] = self.get_f(dag[-1][0].name) + c[0] = torch.sigmoid(self.w_xc(x) + F.linear(h_prev, self.w_hc, None)) + h[0] = (c[0]*f[0](self.w_xh(x) + F.linear(h_prev, self.w_hh, None)) + + (1 - c[0])*h_prev) + + leaf_node_ids = [] + q = collections.deque() + q.append(0) + + # Computes connections from the parent nodes `node_id` + # to their child nodes `next_id` recursively, skipping leaf nodes. A + # leaf node is a node whose id == `self.num_blocks`. + # + # Connections between parent i and child j should be computed as + # h_j = c_j*f_{ij}{(W^h_{ij}*h_i)} + (1 - c_j)*h_i, + # where c_j = \sigmoid{(W^c_{ij}*h_i)} + # + # See Training details from Section 3.1 of the paper. + # + # The following algorithm does a breadth-first (since `q.popleft()` is + # used) search over the nodes and computes all the hidden states. + while True: + if len(q) == 0: + break + + node_id = q.popleft() + nodes = dag[node_id] + + for next_node in nodes: + next_id = next_node.id + if next_id == self.num_blocks: + leaf_node_ids.append(node_id) + assert len(nodes) == 1, ('parent of leaf node should have ' + 'only one child') + continue + + w_h = self.w_h[node_id][next_id] + w_c = self.w_c[node_id][next_id] + + f[next_id] = self.get_f(next_node.name) + c[next_id] = torch.sigmoid(w_c(h[node_id])) + h[next_id] = (c[next_id]*f[next_id](w_h(h[node_id])) + + (1 - c[next_id])*h[node_id]) + + q.append(next_id) + + # Instead of averaging loose ends, perhaps there should + # be a set of separate unshared weights for each "loose" connection + # between each node in a cell and the output. + # + # As it stands, all weights W^h_{ij} are doing double duty by + # connecting both from i to j, as well as from i to the output. + + # average all the loose ends + leaf_nodes = [h[node_id] for node_id in leaf_node_ids] + output = torch.mean(torch.stack(leaf_nodes, 2), -1) + + # stabilizing the Updates of omega + if self.batch_norm is not None: + output = self.batch_norm(output) + + return output, h[self.num_blocks - 1] + + def init_hidden(self, batch_size): + zeros = torch.zeros(batch_size, self.shared_hid) + return utils.get_variable(zeros, self.use_cuda, requires_grad=False) + + def get_f(self, name): + name = name.lower() + if name == 'relu': + f = torch.relu + elif name == 'tanh': + f = torch.tanh + elif name == 'identity': + f = lambda x: x + elif name == 'sigmoid': + f = torch.sigmoid + return f + + + @property + def num_parameters(self): + def size(p): + return np.prod(p.size()) + return sum([size(param) for param in self.parameters()]) + + + def reset_parameters(self): + init_range = 0.025 + # init_range = 0.025 if self.args.mode == 'train' else 0.04 + for param in self.parameters(): + param.data.uniform_(-init_range, init_range) + self.decoder.bias.data.fill_(0) + + def predict(self, word_seq): + """ + + :param word_seq: torch.LongTensor, [batch_size, seq_len] + :return predict: dict of torch.LongTensor, [batch_size, seq_len] + """ + output = self(word_seq) + _, predict = output['pred'].max(dim=1) + return {'pred': predict} diff --git a/legacy/automl/enas_trainer.py b/legacy/automl/enas_trainer.py new file mode 100644 index 00000000..e3524aa9 --- /dev/null +++ b/legacy/automl/enas_trainer.py @@ -0,0 +1,383 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch + +import math +import time +from datetime import datetime +from datetime import timedelta + +import numpy as np +import torch + +try: + from tqdm.auto import tqdm +except: + from fastNLP.core.utils import _pseudo_tqdm as tqdm + +from fastNLP.core.batch import Batch +from fastNLP.core.callback import CallbackException +from fastNLP.core.dataset import DataSet +from fastNLP.core.utils import _move_dict_value_to_device +import fastNLP +from . import enas_utils as utils +from fastNLP.core.utils import _build_args + +from torch.optim import Adam + + +def _get_no_grad_ctx_mgr(): + """Returns a the `torch.no_grad` context manager for PyTorch version >= + 0.4, or a no-op context manager otherwise. + """ + return torch.no_grad() + + +class ENASTrainer(fastNLP.Trainer): + """A class to wrap training code.""" + def __init__(self, train_data, model, controller, **kwargs): + """Constructor for training algorithm. + :param DataSet train_data: the training data + :param torch.nn.modules.module model: a PyTorch model + :param torch.nn.modules.module controller: a PyTorch model + """ + self.final_epochs = kwargs['final_epochs'] + kwargs.pop('final_epochs') + super(ENASTrainer, self).__init__(train_data, model, **kwargs) + self.controller_step = 0 + self.shared_step = 0 + self.max_length = 35 + + self.shared = model + self.controller = controller + + self.shared_optim = Adam( + self.shared.parameters(), + lr=20.0, + weight_decay=1e-7) + + self.controller_optim = Adam( + self.controller.parameters(), + lr=3.5e-4) + + def train(self, load_best_model=True): + """ + :param bool load_best_model: 该参数只有在初始化提供了dev_data的情况下有效，如果True, trainer将在返回之前重新加载dev表现 + 最好的模型参数。 + :return results: 返回一个字典类型的数据, + 内含以下内容:: + + seconds: float, 表示训练时长 + 以下三个内容只有在提供了dev_data的情况下会有。 + best_eval: Dict of Dict, 表示evaluation的结果 + best_epoch: int，在第几个epoch取得的最佳值 + best_step: int, 在第几个step(batch)更新取得的最佳值 + + """ + results = {} + if self.n_epochs <= 0: + print(f"training epoch is {self.n_epochs}, nothing was done.") + results['seconds'] = 0. + return results + try: + if torch.cuda.is_available() and self.use_cuda: + self.model = self.model.cuda() + self._model_device = self.model.parameters().__next__().device + self._mode(self.model, is_test=False) + + self.start_time = str(datetime.now().strftime('%Y-%m-%d-%H-%M-%S')) + start_time = time.time() + print("training epochs started " + self.start_time, flush=True) + + try: + self.callback_manager.on_train_begin() + self._train() + self.callback_manager.on_train_end(self.model) + except (CallbackException, KeyboardInterrupt) as e: + self.callback_manager.on_exception(e, self.model) + + if self.dev_data is not None: + print("\nIn Epoch:{}/Step:{}, got best dev performance:".format(self.best_dev_epoch, self.best_dev_step) + + self.tester._format_eval_results(self.best_dev_perf),) + results['best_eval'] = self.best_dev_perf + results['best_epoch'] = self.best_dev_epoch + results['best_step'] = self.best_dev_step + if load_best_model: + model_name = "best_" + "_".join([self.model.__class__.__name__, self.metric_key, self.start_time]) + load_succeed = self._load_model(self.model, model_name) + if load_succeed: + print("Reloaded the best model.") + else: + print("Fail to reload best model.") + finally: + pass + results['seconds'] = round(time.time() - start_time, 2) + + return results + + def _train(self): + if not self.use_tqdm: + from fastNLP.core.utils import _pseudo_tqdm as inner_tqdm + else: + inner_tqdm = tqdm + self.step = 0 + start = time.time() + total_steps = (len(self.train_data) // self.batch_size + int( + len(self.train_data) % self.batch_size != 0)) * self.n_epochs + with inner_tqdm(total=total_steps, postfix='loss:{0:<6.5f}', leave=False, dynamic_ncols=True) as pbar: + avg_loss = 0 + data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + for epoch in range(1, self.n_epochs+1): + pbar.set_description_str(desc="Epoch {}/{}".format(epoch, self.n_epochs)) + last_stage = (epoch > self.n_epochs + 1 - self.final_epochs) + if epoch == self.n_epochs + 1 - self.final_epochs: + print('Entering the final stage. (Only train the selected structure)') + # early stopping + self.callback_manager.on_epoch_begin(epoch, self.n_epochs) + + # 1. Training the shared parameters omega of the child models + self.train_shared(pbar) + + # 2. Training the controller parameters theta + if not last_stage: + self.train_controller() + + if ((self.validate_every > 0 and self.step % self.validate_every == 0) or + (self.validate_every < 0 and self.step % len(data_iterator) == 0)) \ + and self.dev_data is not None: + if not last_stage: + self.derive() + eval_res = self._do_validation(epoch=epoch, step=self.step) + eval_str = "Evaluation at Epoch {}/{}. Step:{}/{}. ".format(epoch, self.n_epochs, self.step, + total_steps) + \ + self.tester._format_eval_results(eval_res) + pbar.write(eval_str) + + # lr decay; early stopping + self.callback_manager.on_epoch_end(epoch, self.n_epochs, self.optimizer) + # =============== epochs end =================== # + pbar.close() + # ============ tqdm end ============== # + + + def get_loss(self, inputs, targets, hidden, dags): + """Computes the loss for the same batch for M models. + + This amounts to an estimate of the loss, which is turned into an + estimate for the gradients of the shared model. + """ + if not isinstance(dags, list): + dags = [dags] + + loss = 0 + for dag in dags: + self.shared.setDAG(dag) + inputs = _build_args(self.shared.forward, **inputs) + inputs['hidden'] = hidden + result = self.shared(**inputs) + output, hidden, extra_out = result['pred'], result['hidden'], result['extra_out'] + + self.callback_manager.on_loss_begin(targets, result) + sample_loss = self._compute_loss(result, targets) + loss += sample_loss + + assert len(dags) == 1, 'there are multiple `hidden` for multple `dags`' + return loss, hidden, extra_out + + def train_shared(self, pbar=None, max_step=None, dag=None): + """Train the language model for 400 steps of minibatches of 64 + examples. + + Args: + max_step: Used to run extra training steps as a warm-up. + dag: If not None, is used instead of calling sample(). + + BPTT is truncated at 35 timesteps. + + For each weight update, gradients are estimated by sampling M models + from the fixed controller policy, and averaging their gradients + computed on a batch of training data. + """ + model = self.shared + model.train() + self.controller.eval() + + hidden = self.shared.init_hidden(self.batch_size) + + abs_max_grad = 0 + abs_max_hidden_norm = 0 + step = 0 + raw_total_loss = 0 + total_loss = 0 + train_idx = 0 + avg_loss = 0 + data_iterator = Batch(self.train_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + + for batch_x, batch_y in data_iterator: + _move_dict_value_to_device(batch_x, batch_y, device=self._model_device) + indices = data_iterator.get_batch_indices() + # negative sampling; replace unknown; re-weight batch_y + self.callback_manager.on_batch_begin(batch_x, batch_y, indices) + # prediction = self._data_forward(self.model, batch_x) + + dags = self.controller.sample(1) + inputs, targets = batch_x, batch_y + # self.callback_manager.on_loss_begin(batch_y, prediction) + loss, hidden, extra_out = self.get_loss(inputs, + targets, + hidden, + dags) + hidden.detach_() + + avg_loss += loss.item() + + # Is loss NaN or inf? requires_grad = False + self.callback_manager.on_backward_begin(loss, self.model) + self._grad_backward(loss) + self.callback_manager.on_backward_end(self.model) + + self._update() + self.callback_manager.on_step_end(self.optimizer) + + if (self.step+1) % self.print_every == 0: + if self.use_tqdm: + print_output = "loss:{0:<6.5f}".format(avg_loss / self.print_every) + pbar.update(self.print_every) + else: + end = time.time() + diff = timedelta(seconds=round(end - start)) + print_output = "[epoch: {:>3} step: {:>4}] train loss: {:>4.6} time: {}".format( + epoch, self.step, avg_loss, diff) + pbar.set_postfix_str(print_output) + avg_loss = 0 + self.step += 1 + step += 1 + self.shared_step += 1 + self.callback_manager.on_batch_end() + # ================= mini-batch end ==================== # + + + def get_reward(self, dag, entropies, hidden, valid_idx=0): + """Computes the perplexity of a single sampled model on a minibatch of + validation data. + """ + if not isinstance(entropies, np.ndarray): + entropies = entropies.data.cpu().numpy() + + data_iterator = Batch(self.dev_data, batch_size=self.batch_size, sampler=self.sampler, as_numpy=False, + prefetch=self.prefetch) + + for inputs, targets in data_iterator: + valid_loss, hidden, _ = self.get_loss(inputs, targets, hidden, dag) + valid_loss = utils.to_item(valid_loss.data) + + valid_ppl = math.exp(valid_loss) + + R = 80 / valid_ppl + + rewards = R + 1e-4 * entropies + + return rewards, hidden + + def train_controller(self): + """Fixes the shared parameters and updates the controller parameters. + + The controller is updated with a score function gradient estimator + (i.e., REINFORCE), with the reward being c/valid_ppl, where valid_ppl + is computed on a minibatch of validation data. + + A moving average baseline is used. + + The controller is trained for 2000 steps per epoch (i.e., + first (Train Shared) phase -> second (Train Controller) phase). + """ + model = self.controller + model.train() + # Why can't we call shared.eval() here? Leads to loss + # being uniformly zero for the controller. + # self.shared.eval() + + avg_reward_base = None + baseline = None + adv_history = [] + entropy_history = [] + reward_history = [] + + hidden = self.shared.init_hidden(self.batch_size) + total_loss = 0 + valid_idx = 0 + for step in range(20): + # sample models + dags, log_probs, entropies = self.controller.sample( + with_details=True) + + # calculate reward + np_entropies = entropies.data.cpu().numpy() + # No gradients should be backpropagated to the + # shared model during controller training, obviously. + with _get_no_grad_ctx_mgr(): + rewards, hidden = self.get_reward(dags, + np_entropies, + hidden, + valid_idx) + + + reward_history.extend(rewards) + entropy_history.extend(np_entropies) + + # moving average baseline + if baseline is None: + baseline = rewards + else: + decay = 0.95 + baseline = decay * baseline + (1 - decay) * rewards + + adv = rewards - baseline + adv_history.extend(adv) + + # policy loss + loss = -log_probs*utils.get_variable(adv, + self.use_cuda, + requires_grad=False) + + loss = loss.sum() # or loss.mean() + + # update + self.controller_optim.zero_grad() + loss.backward() + + self.controller_optim.step() + + total_loss += utils.to_item(loss.data) + + if ((step % 50) == 0) and (step > 0): + reward_history, adv_history, entropy_history = [], [], [] + total_loss = 0 + + self.controller_step += 1 + # prev_valid_idx = valid_idx + # valid_idx = ((valid_idx + self.max_length) % + # (self.valid_data.size(0) - 1)) + # # Whenever we wrap around to the beginning of the + # # validation data, we reset the hidden states. + # if prev_valid_idx > valid_idx: + # hidden = self.shared.init_hidden(self.batch_size) + + def derive(self, sample_num=10, valid_idx=0): + """We are always deriving based on the very first batch + of validation data? This seems wrong... + """ + hidden = self.shared.init_hidden(self.batch_size) + + dags, _, entropies = self.controller.sample(sample_num, + with_details=True) + + max_R = 0 + best_dag = None + for dag in dags: + R, _ = self.get_reward(dag, entropies, hidden, valid_idx) + if R.max() > max_R: + max_R = R.max() + best_dag = dag + + self.model.setDAG(best_dag) diff --git a/legacy/automl/enas_utils.py b/legacy/automl/enas_utils.py new file mode 100644 index 00000000..7a53dd12 --- /dev/null +++ b/legacy/automl/enas_utils.py @@ -0,0 +1,53 @@ +# Code Modified from https://github.com/carpedm20/ENAS-pytorch + +from __future__ import print_function + +import collections +from collections import defaultdict + +import numpy as np +import torch +from torch.autograd import Variable + + +def detach(h): + if type(h) == Variable: + return Variable(h.data) + else: + return tuple(detach(v) for v in h) + +def get_variable(inputs, cuda=False, **kwargs): + if type(inputs) in [list, np.ndarray]: + inputs = torch.Tensor(inputs) + if cuda: + out = Variable(inputs.cuda(), **kwargs) + else: + out = Variable(inputs, **kwargs) + return out + +def update_lr(optimizer, lr): + for param_group in optimizer.param_groups: + param_group['lr'] = lr + +Node = collections.namedtuple('Node', ['id', 'name']) + + +class keydefaultdict(defaultdict): + def __missing__(self, key): + if self.default_factory is None: + raise KeyError(key) + else: + ret = self[key] = self.default_factory(key) + return ret + + +def to_item(x): + """Converts x, possibly scalar and possibly tensor, to a Python scalar.""" + if isinstance(x, (float, int)): + return x + + if float(torch.__version__[0:3]) < 0.4: + assert (x.dim() == 1) and (len(x) == 1) + return x[0] + + return x.item() diff --git a/legacy/component/__init__.py b/legacy/component/__init__.py new file mode 100644 index 00000000..c6784aef --- /dev/null +++ b/legacy/component/__init__.py @@ -0,0 +1 @@ +from .bert_tokenizer import BertTokenizer diff --git a/legacy/component/bert_tokenizer.py b/legacy/component/bert_tokenizer.py new file mode 100644 index 00000000..6354076d --- /dev/null +++ b/legacy/component/bert_tokenizer.py @@ -0,0 +1,378 @@ +""" +bert_tokenizer.py is modified from huggingface/pytorch-pretrained-BERT, which is licensed under the Apache License 2.0. +""" +import collections +import os +import unicodedata +from io import open + + +PRETRAINED_VOCAB_ARCHIVE_MAP = { + 'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt", + 'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt", + 'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt", + 'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txt", + 'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-vocab.txt", + 'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt", + 'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt", +} +PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP = { + 'bert-base-uncased': 512, + 'bert-large-uncased': 512, + 'bert-base-cased': 512, + 'bert-large-cased': 512, + 'bert-base-multilingual-uncased': 512, + 'bert-base-multilingual-cased': 512, + 'bert-base-chinese': 512, +} +VOCAB_NAME = 'vocab.txt' + + +def load_vocab(vocab_file): + """Loads a vocabulary file into a dictionary.""" + vocab = collections.OrderedDict() + index = 0 + with open(vocab_file, "r", encoding="utf-8") as reader: + while True: + token = reader.readline() + if not token: + break + token = token.strip() + vocab[token] = index + index += 1 + return vocab + + +def whitespace_tokenize(text): + """Runs basic whitespace cleaning and splitting on a piece of text.""" + text = text.strip() + if not text: + return [] + tokens = text.split() + return tokens + + +class BertTokenizer(object): + """Runs end-to-end tokenization: punctuation splitting + wordpiece""" + + def __init__(self, vocab_file, do_lower_case=True, max_len=None, do_basic_tokenize=True, + never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")): + """Constructs a BertTokenizer. + Args: + vocab_file: Path to a one-wordpiece-per-line vocabulary file + do_lower_case: Whether to lower case the input + Only has an effect when do_wordpiece_only=False + do_basic_tokenize: Whether to do basic tokenization before wordpiece. + max_len: An artificial maximum length to truncate tokenized sequences to; + Effective maximum length is always the minimum of this + value (if specified) and the underlying BERT model's + sequence length. + never_split: List of tokens which will never be split during tokenization. + Only has an effect when do_wordpiece_only=False + """ + if not os.path.isfile(vocab_file): + raise ValueError( + "Can't find a vocabulary file at path '{}'. To load the vocabulary from a Google pretrained " + "model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`".format(vocab_file)) + self.vocab = load_vocab(vocab_file) + self.ids_to_tokens = collections.OrderedDict( + [(ids, tok) for tok, ids in self.vocab.items()]) + self.do_basic_tokenize = do_basic_tokenize + if do_basic_tokenize: + self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case, + never_split=never_split) + self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab) + self.max_len = max_len if max_len is not None else int(1e12) + + def tokenize(self, text): + split_tokens = [] + if self.do_basic_tokenize: + for token in self.basic_tokenizer.tokenize(text): + for sub_token in self.wordpiece_tokenizer.tokenize(token): + split_tokens.append(sub_token) + else: + split_tokens = self.wordpiece_tokenizer.tokenize(text) + return split_tokens + + def convert_tokens_to_ids(self, tokens): + """Converts a sequence of tokens into ids using the vocab.""" + ids = [] + for token in tokens: + ids.append(self.vocab[token]) + if len(ids) > self.max_len: + print( + "WARNING!\n\"" + "Token indices sequence length is longer than the specified maximum " + "sequence length for this BERT model ({} > {}). Running this" + " sequence through BERT will result in indexing errors".format(len(ids), self.max_len) + ) + return ids + + def convert_ids_to_tokens(self, ids): + """Converts a sequence of ids in wordpiece tokens using the vocab.""" + tokens = [] + for i in ids: + tokens.append(self.ids_to_tokens[i]) + return tokens + + def save_vocabulary(self, vocab_path): + """Save the tokenizer vocabulary to a directory or file.""" + index = 0 + if os.path.isdir(vocab_path): + vocab_file = os.path.join(vocab_path, VOCAB_NAME) + with open(vocab_file, "w", encoding="utf-8") as writer: + for token, token_index in sorted(self.vocab.items(), key=lambda kv: kv[1]): + if index != token_index: + print("Saving vocabulary to {}: vocabulary indices are not consecutive." + " Please check that the vocabulary is not corrupted!".format(vocab_file)) + index = token_index + writer.write(token + u'\n') + index += 1 + return vocab_file + + @classmethod + def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs): + """ + Instantiate a PreTrainedBertModel from a pre-trained model file. + Download and cache the pre-trained model file if needed. + """ + if pretrained_model_name_or_path in PRETRAINED_VOCAB_ARCHIVE_MAP: + vocab_file = PRETRAINED_VOCAB_ARCHIVE_MAP[pretrained_model_name_or_path] + if '-cased' in pretrained_model_name_or_path and kwargs.get('do_lower_case', True): + print("The pre-trained model you are loading is a cased model but you have not set " + "`do_lower_case` to False. We are setting `do_lower_case=False` for you but " + "you may want to check this behavior.") + kwargs['do_lower_case'] = False + elif '-cased' not in pretrained_model_name_or_path and not kwargs.get('do_lower_case', True): + print("The pre-trained model you are loading is an uncased model but you have set " + "`do_lower_case` to False. We are setting `do_lower_case=True` for you " + "but you may want to check this behavior.") + kwargs['do_lower_case'] = True + else: + vocab_file = pretrained_model_name_or_path + if os.path.isdir(vocab_file): + vocab_file = os.path.join(vocab_file, VOCAB_NAME) + # redirect to the cache, if necessary + resolved_vocab_file = vocab_file + print("loading vocabulary file {}".format(vocab_file)) + if pretrained_model_name_or_path in PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP: + # if we're using a pretrained model, ensure the tokenizer wont index sequences longer + # than the number of positional embeddings + max_len = PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP[pretrained_model_name_or_path] + kwargs['max_len'] = min(kwargs.get('max_len', int(1e12)), max_len) + # Instantiate tokenizer. + tokenizer = cls(resolved_vocab_file, *inputs, **kwargs) + return tokenizer + + +class BasicTokenizer(object): + """Runs basic tokenization (punctuation splitting, lower casing, etc.).""" + + def __init__(self, + do_lower_case=True, + never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")): + """Constructs a BasicTokenizer. + Args: + do_lower_case: Whether to lower case the input. + """ + self.do_lower_case = do_lower_case + self.never_split = never_split + + def tokenize(self, text): + """Tokenizes a piece of text.""" + text = self._clean_text(text) + # This was added on November 1st, 2018 for the multilingual and Chinese + # models. This is also applied to the English models now, but it doesn't + # matter since the English models were not trained on any Chinese data + # and generally don't have any Chinese data in them (there are Chinese + # characters in the vocabulary because Wikipedia does have some Chinese + # words in the English Wikipedia.). + text = self._tokenize_chinese_chars(text) + orig_tokens = whitespace_tokenize(text) + split_tokens = [] + for token in orig_tokens: + if self.do_lower_case and token not in self.never_split: + token = token.lower() + token = self._run_strip_accents(token) + split_tokens.extend(self._run_split_on_punc(token)) + + output_tokens = whitespace_tokenize(" ".join(split_tokens)) + return output_tokens + + def _run_strip_accents(self, text): + """Strips accents from a piece of text.""" + text = unicodedata.normalize("NFD", text) + output = [] + for char in text: + cat = unicodedata.category(char) + if cat == "Mn": + continue + output.append(char) + return "".join(output) + + def _run_split_on_punc(self, text): + """Splits punctuation on a piece of text.""" + if text in self.never_split: + return [text] + chars = list(text) + i = 0 + start_new_word = True + output = [] + while i < len(chars): + char = chars[i] + if _is_punctuation(char): + output.append([char]) + start_new_word = True + else: + if start_new_word: + output.append([]) + start_new_word = False + output[-1].append(char) + i += 1 + + return ["".join(x) for x in output] + + def _tokenize_chinese_chars(self, text): + """Adds whitespace around any CJK character.""" + output = [] + for char in text: + cp = ord(char) + if self._is_chinese_char(cp): + output.append(" ") + output.append(char) + output.append(" ") + else: + output.append(char) + return "".join(output) + + def _is_chinese_char(self, cp): + """Checks whether CP is the codepoint of a CJK character.""" + # This defines a "chinese character" as anything in the CJK Unicode block: + # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) + # + # Note that the CJK Unicode block is NOT all Japanese and Korean characters, + # despite its name. The modern Korean Hangul alphabet is a different block, + # as is Japanese Hiragana and Katakana. Those alphabets are used to write + # space-separated words, so they are not treated specially and handled + # like the all of the other languages. + if ((cp >= 0x4E00 and cp <= 0x9FFF) or # + (cp >= 0x3400 and cp <= 0x4DBF) or # + (cp >= 0x20000 and cp <= 0x2A6DF) or # + (cp >= 0x2A700 and cp <= 0x2B73F) or # + (cp >= 0x2B740 and cp <= 0x2B81F) or # + (cp >= 0x2B820 and cp <= 0x2CEAF) or + (cp >= 0xF900 and cp <= 0xFAFF) or # + (cp >= 0x2F800 and cp <= 0x2FA1F)): # + return True + + return False + + def _clean_text(self, text): + """Performs invalid character removal and whitespace cleanup on text.""" + output = [] + for char in text: + cp = ord(char) + if cp == 0 or cp == 0xfffd or _is_control(char): + continue + if _is_whitespace(char): + output.append(" ") + else: + output.append(char) + return "".join(output) + + +class WordpieceTokenizer(object): + """Runs WordPiece tokenization.""" + + def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=100): + self.vocab = vocab + self.unk_token = unk_token + self.max_input_chars_per_word = max_input_chars_per_word + + def tokenize(self, text): + """Tokenizes a piece of text into its word pieces. + This uses a greedy longest-match-first algorithm to perform tokenization + using the given vocabulary. + For example: + input = "unaffable" + output = ["un", "##aff", "##able"] + Args: + text: A single token or whitespace separated tokens. This should have + already been passed through `BasicTokenizer`. + Returns: + A list of wordpiece tokens. + """ + + output_tokens = [] + for token in whitespace_tokenize(text): + chars = list(token) + if len(chars) > self.max_input_chars_per_word: + output_tokens.append(self.unk_token) + continue + + is_bad = False + start = 0 + sub_tokens = [] + while start < len(chars): + end = len(chars) + cur_substr = None + while start < end: + substr = "".join(chars[start:end]) + if start > 0: + substr = "##" + substr + if substr in self.vocab: + cur_substr = substr + break + end -= 1 + if cur_substr is None: + is_bad = True + break + sub_tokens.append(cur_substr) + start = end + + if is_bad: + output_tokens.append(self.unk_token) + else: + output_tokens.extend(sub_tokens) + return output_tokens + + +def _is_whitespace(char): + """Checks whether `chars` is a whitespace character.""" + # \t, \n, and \r are technically contorl characters but we treat them + # as whitespace since they are generally considered as such. + if char == " " or char == "\t" or char == "\n" or char == "\r": + return True + cat = unicodedata.category(char) + if cat == "Zs": + return True + return False + + +def _is_control(char): + """Checks whether `chars` is a control character.""" + # These are technically control characters but we count them as whitespace + # characters. + if char == "\t" or char == "\n" or char == "\r": + return False + cat = unicodedata.category(char) + if cat.startswith("C"): + return True + return False + + +def _is_punctuation(char): + """Checks whether `chars` is a punctuation character.""" + cp = ord(char) + # We treat all non-letter/number ASCII as punctuation. + # Characters such as "^", "$", and "`" are not in the Unicode + # Punctuation class but we treat them as punctuation anyways, for + # consistency. + if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or + (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)): + return True + cat = unicodedata.category(char) + if cat.startswith("P"): + return True + return False + diff --git a/readthedocs.yml b/readthedocs.yml index 9b172987..e6d5bafd 100644 --- a/readthedocs.yml +++ b/readthedocs.yml @@ -1,6 +1,16 @@ +version: 2 + +sphinx: + configuration: docs/source/conf.py + build: image: latest python: version: 3.6 - setup_py_install: true \ No newline at end of file + install: + - method: setuptools + path: . + +formats: + - htmlzip \ No newline at end of file diff --git a/reproduction/Biaffine_parser/cfg.cfg b/reproduction/Biaffine_parser/cfg.cfg index 9b00c209..03040600 100644 --- a/reproduction/Biaffine_parser/cfg.cfg +++ b/reproduction/Biaffine_parser/cfg.cfg @@ -1,8 +1,9 @@ [train] -n_epochs = 40 +n_epochs = 20 batch_size = 32 use_cuda = true -validate_every = 500 +use_tqdm=true +validate_every = 1000 use_golden_train=true [test] @@ -16,20 +17,18 @@ use_cuda = true [model] word_vocab_size = -1 -word_emb_dim = 100 +word_emb_dim = 300 pos_vocab_size = -1 pos_emb_dim = 100 -word_hid_dim = 100 -pos_hid_dim = 100 rnn_layers = 3 -rnn_hidden_size = 400 +rnn_hidden_size = 256 arc_mlp_size = 500 label_mlp_size = 100 num_label = -1 -dropout = 0.33 -use_var_lstm=true +dropout = 0.3 +encoder="var-lstm" use_greedy_infer=false [optim] -lr = 3e-4 +lr = 2e-3 ;weight_decay = 3e-5 diff --git a/reproduction/Biaffine_parser/main.py b/reproduction/Biaffine_parser/main.py index 9028ff80..f4fd5836 100644 --- a/reproduction/Biaffine_parser/main.py +++ b/reproduction/Biaffine_parser/main.py @@ -5,7 +5,7 @@ sys.path.extend(['/home/yfshao/workdir/dev_fastnlp']) import torch import argparse -from reproduction.Biaffine_parser.util import ConllxDataLoader, add_seg_tag +from fastNLP.io.dataset_loader import ConllxDataLoader, add_seg_tag from fastNLP.core.dataset import DataSet from fastNLP.core.instance import Instance diff --git a/reproduction/Biaffine_parser/run.py b/reproduction/Biaffine_parser/run.py index 656da201..a69d3d58 100644 --- a/reproduction/Biaffine_parser/run.py +++ b/reproduction/Biaffine_parser/run.py @@ -4,25 +4,23 @@ import sys sys.path.append(os.path.join(os.path.dirname(__file__), '../..')) import fastNLP -import torch from fastNLP.core.trainer import Trainer from fastNLP.core.instance import Instance from fastNLP.api.pipeline import Pipeline from fastNLP.models.biaffine_parser import BiaffineParser, ParserMetric, ParserLoss -from fastNLP.core.vocabulary import Vocabulary -from fastNLP.core.dataset import DataSet from fastNLP.core.tester import Tester from fastNLP.io.config_io import ConfigLoader, ConfigSection from fastNLP.io.model_io import ModelLoader -from fastNLP.io.embed_loader import EmbedLoader -from fastNLP.io.model_io import ModelSaver -from reproduction.Biaffine_parser.util import ConllxDataLoader, MyDataloader +from fastNLP.io.dataset_loader import ConllxDataLoader from fastNLP.api.processor import * +from fastNLP.io.embed_loader import EmbedLoader +from fastNLP.core.callback import Callback BOS = '' EOS = '' UNK = '' +PAD = '' NUM = '' ENG = '' @@ -33,11 +31,11 @@ if len(os.path.dirname(__file__)) != 0: def convert(data): dataset = DataSet() for sample in data: - word_seq = [BOS] + sample[0] - pos_seq = [BOS] + sample[1] - heads = [0] + list(map(int, sample[2])) - head_tags = [BOS] + sample[3] - dataset.append(Instance(words=word_seq, + word_seq = [BOS] + sample['words'] + pos_seq = [BOS] + sample['pos_tags'] + heads = [0] + sample['heads'] + head_tags = [BOS] + sample['labels'] + dataset.append(Instance(raw_words=word_seq, pos=pos_seq, gold_heads=heads, arc_true=heads, @@ -50,24 +48,11 @@ def load(path): return convert(data) -# datadir = "/mnt/c/Me/Dev/release-2.2-st-train-dev-data/ud-treebanks-v2.2/UD_English-EWT" -# datadir = "/home/yfshao/UD_English-EWT" -# train_data_name = "en_ewt-ud-train.conllu" -# dev_data_name = "en_ewt-ud-dev.conllu" -# emb_file_name = '/home/yfshao/glove.6B.100d.txt' -# loader = ConlluDataLoader() - -# datadir = '/home/yfshao/workdir/parser-data/' -# train_data_name = "train_ctb5.txt" -# dev_data_name = "dev_ctb5.txt" -# test_data_name = "test_ctb5.txt" - -datadir = "/home/yfshao/workdir/ctb7.0/" +datadir = "/remote-home/yfshao/workdir/ctb9.0/" train_data_name = "train.conllx" dev_data_name = "dev.conllx" test_data_name = "test.conllx" -# emb_file_name = "/home/yfshao/workdir/parser-data/word_OOVthr_30_100v.txt" -emb_file_name = "/home/yfshao/workdir/word_vector/cc.zh.300.vec" +emb_file_name = "/remote-home/yfshao/workdir/word_vector/cc.zh.300.vec" cfgfile = './cfg.cfg' processed_datadir = './save' @@ -113,27 +98,23 @@ def update_v(vocab, data, field): data.apply(lambda x: vocab.add_word_lst(x[field]), new_field_name=None) -print('load raw data and preprocess') # use pretrain embedding -word_v = Vocabulary() -word_v.unknown_label = UNK -pos_v = Vocabulary() +word_v = Vocabulary(unknown=UNK, padding=PAD) +pos_v = Vocabulary(unknown=None, padding=PAD) tag_v = Vocabulary(unknown=None, padding=None) train_data = load(os.path.join(datadir, train_data_name)) dev_data = load(os.path.join(datadir, dev_data_name)) test_data = load(os.path.join(datadir, test_data_name)) -print(train_data[0]) -num_p = Num2TagProcessor('words', 'words') +print('load raw data and preprocess') + +num_p = Num2TagProcessor(tag=NUM, field_name='raw_words', new_added_field_name='words') for ds in (train_data, dev_data, test_data): num_p(ds) - update_v(word_v, train_data, 'words') update_v(pos_v, train_data, 'pos') update_v(tag_v, train_data, 'tags') print('vocab build success {}, {}, {}'.format(len(word_v), len(pos_v), len(tag_v))) -# embed, _ = EmbedLoader.fast_load_embedding(model_args['word_emb_dim'], emb_file_name, word_v) -# print(embed.size()) # Model model_args['word_vocab_size'] = len(word_v) @@ -141,7 +122,7 @@ model_args['pos_vocab_size'] = len(pos_v) model_args['num_label'] = len(tag_v) model = BiaffineParser(**model_args.data) -model.reset_parameters() +print(model) word_idxp = IndexerProcessor(word_v, 'words', 'word_seq') pos_idxp = IndexerProcessor(pos_v, 'pos', 'pos_seq') @@ -164,7 +145,6 @@ for ds in (train_data, dev_data, test_data): if train_args['use_golden_train']: train_data.set_input('gold_heads', flag=True) train_args.data.pop('use_golden_train') -ignore_label = pos_v['punct'] print(test_data[0]) print('train len {}'.format(len(train_data))) @@ -172,44 +152,62 @@ print('dev len {}'.format(len(dev_data))) print('test len {}'.format(len(test_data))) - def train(path): # test saving pipeline save_pipe(path) + embed = EmbedLoader.load_with_vocab(emb_file_name, word_v) + embed = torch.tensor(embed, dtype=torch.float32) - # Trainer - trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, - loss=ParserLoss(), metrics=ParserMetric(), metric_key='UAS', - **train_args.data, - optimizer=fastNLP.Adam(**optim_args.data), - save_path=path) - - # model.word_embedding = torch.nn.Embedding.from_pretrained(embed, freeze=False) + # embed = EmbedLoader.fast_load_embedding(emb_dim=model_args['word_emb_dim'], emb_file=emb_file_name, vocab=word_v) + # embed = torch.tensor(embed, dtype=torch.float32) + # model.word_embedding = torch.nn.Embedding.from_pretrained(embed, freeze=True) model.word_embedding.padding_idx = word_v.padding_idx model.word_embedding.weight.data[word_v.padding_idx].fill_(0) model.pos_embedding.padding_idx = pos_v.padding_idx model.pos_embedding.weight.data[pos_v.padding_idx].fill_(0) - # try: - # ModelLoader.load_pytorch(model, "./save/saved_model.pkl") - # print('model parameter loaded!') - # except Exception as _: - # print("No saved model. Continue.") - # pass + class MyCallback(Callback): + def on_step_end(self, optimizer): + step = self.trainer.step + # learning rate decay + if step > 0 and step % 1000 == 0: + for pg in optimizer.param_groups: + pg['lr'] *= 0.93 + print('decay lr to {}'.format([pg['lr'] for pg in optimizer.param_groups])) + + if step == 3000: + # start training embedding + print('start training embedding at {}'.format(step)) + model = self.trainer.model + for m in model.modules(): + if isinstance(m, torch.nn.Embedding): + m.weight.requires_grad = True - # Start training - trainer.train() - print("Training finished!") + # Trainer + trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data, + loss=ParserLoss(), metrics=ParserMetric(), metric_key='UAS', + **train_args.data, + optimizer=fastNLP.Adam(**optim_args.data), + save_path=path, + callbacks=[MyCallback()]) - # save pipeline - save_pipe(path) - print('pipe saved') + # Start training + try: + trainer.train() + print("Training finished!") + finally: + # save pipeline + save_pipe(path) + print('pipe saved') def save_pipe(path): pipe = Pipeline(processors=[num_p, word_idxp, pos_idxp, seq_p, set_input_p]) pipe.add_processor(ModelProcessor(model=model, batch_size=32)) pipe.add_processor(label_toword_p) - torch.save(pipe, os.path.join(path, 'pipe.pkl')) + os.makedirs(path, exist_ok=True) + torch.save({'pipeline': pipe, + 'names':['num word_idx pos_idx seq set_input model tag_to_word'.split()], + }, os.path.join(path, 'pipe.pkl')) def test(path): @@ -234,16 +232,11 @@ def test(path): print("Testing Test data") tester.test(model, test_data) -def build_pipe(parser_pipe_path): - parser_pipe = torch.load(parser_pipe_path) - - - if __name__ == "__main__": import argparse parser = argparse.ArgumentParser(description='Run a chinese word segmentation model') - parser.add_argument('--mode', help='set the model\'s model', choices=['train', 'test', 'infer', 'save']) + parser.add_argument('--mode', help='set the model\'s model', choices=['train', 'test', 'infer']) parser.add_argument('--path', type=str, default='') # parser.add_argument('--dst', type=str, default='') args = parser.parse_args() @@ -253,12 +246,6 @@ if __name__ == "__main__": test(args.path) elif args.mode == 'infer': pass - # elif args.mode == 'save': - # print(f'save model from {args.path} to {args.dst}') - # save_model(args.path, args.dst) - # load_path = os.path.dirname(args.dst) - # print(f'save pipeline in {load_path}') - # build(load_path) else: print('no mode specified for model!') parser.print_help() diff --git a/reproduction/Biaffine_parser/util.py b/reproduction/Biaffine_parser/util.py index 793b1fb2..aa40e4e9 100644 --- a/reproduction/Biaffine_parser/util.py +++ b/reproduction/Biaffine_parser/util.py @@ -1,34 +1,3 @@ -class ConllxDataLoader(object): - def load(self, path): - datalist = [] - with open(path, 'r', encoding='utf-8') as f: - sample = [] - for line in f: - if line.startswith('\n'): - datalist.append(sample) - sample = [] - elif line.startswith('#'): - continue - else: - sample.append(line.split('\t')) - if len(sample) > 0: - datalist.append(sample) - - data = [self.get_one(sample) for sample in datalist] - return list(filter(lambda x: x is not None, data)) - - def get_one(self, sample): - sample = list(map(list, zip(*sample))) - if len(sample) == 0: - return None - for w in sample[7]: - if w == '_': - print('Error Sample {}'.format(sample)) - return None - # return word_seq, pos_seq, head_seq, head_tag_seq - return sample[1], sample[3], list(map(int, sample[6])), sample[7] - - class MyDataloader: def load(self, data_path): with open(data_path, "r", encoding="utf-8") as f: @@ -56,23 +25,3 @@ class MyDataloader: return data -def add_seg_tag(data): - """ - - :param data: list of ([word], [pos], [heads], [head_tags]) - :return: list of ([word], [pos]) - """ - - _processed = [] - for word_list, pos_list, _, _ in data: - new_sample = [] - for word, pos in zip(word_list, pos_list): - if len(word) == 1: - new_sample.append((word, 'S-' + pos)) - else: - new_sample.append((word[0], 'B-' + pos)) - for c in word[1:-1]: - new_sample.append((c, 'M-' + pos)) - new_sample.append((word[-1], 'E-' + pos)) - _processed.append(list(map(list, zip(*new_sample)))) - return _processed \ No newline at end of file diff --git a/reproduction/chinese_word_segment/models/__init__.py b/reproduction/Chinese_word_segmentation/__init__.py similarity index 100% rename from reproduction/chinese_word_segment/models/__init__.py rename to reproduction/Chinese_word_segmentation/__init__.py diff --git a/reproduction/chinese_word_segment/cws.cfg b/reproduction/Chinese_word_segmentation/cws.cfg similarity index 100% rename from reproduction/chinese_word_segment/cws.cfg rename to reproduction/Chinese_word_segmentation/cws.cfg diff --git a/reproduction/chinese_word_segment/process/__init__.py b/reproduction/Chinese_word_segmentation/cws_io/__init__.py similarity index 100% rename from reproduction/chinese_word_segment/process/__init__.py rename to reproduction/Chinese_word_segmentation/cws_io/__init__.py diff --git a/reproduction/Chinese_word_segmentation/cws_io/cws_reader.py b/reproduction/Chinese_word_segmentation/cws_io/cws_reader.py new file mode 100644 index 00000000..b28b04f6 --- /dev/null +++ b/reproduction/Chinese_word_segmentation/cws_io/cws_reader.py @@ -0,0 +1,3 @@ + + + diff --git a/reproduction/Chinese_word_segmentation/models/__init__.py b/reproduction/Chinese_word_segmentation/models/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/reproduction/chinese_word_segment/models/cws_model.py b/reproduction/Chinese_word_segmentation/models/cws_model.py similarity index 94% rename from reproduction/chinese_word_segment/models/cws_model.py rename to reproduction/Chinese_word_segmentation/models/cws_model.py index c6cf6746..b41ad87d 100644 --- a/reproduction/chinese_word_segment/models/cws_model.py +++ b/reproduction/Chinese_word_segmentation/models/cws_model.py @@ -1,11 +1,11 @@ -from torch import nn import torch -import torch.nn.functional as F +from torch import nn -from fastNLP.modules.decoder.MLP import MLP from fastNLP.models.base_model import BaseModel -from reproduction.chinese_word_segment.utils import seq_lens_to_mask +from fastNLP.modules.decoder.mlp import MLP +from reproduction.Chinese_word_segmentation.utils import seq_lens_to_mask + class CWSBiLSTMEncoder(BaseModel): def __init__(self, vocab_num, embed_dim=100, bigram_vocab_num=None, bigram_embed_dim=100, num_bigram_per_char=None, @@ -120,8 +120,8 @@ class CWSBiLSTMSegApp(BaseModel): return {'pred_tags': pred_tags} -from fastNLP.modules.decoder.CRF import ConditionalRandomField -from fastNLP.modules.decoder.CRF import allowed_transitions +from fastNLP.modules.decoder.crf import ConditionalRandomField +from fastNLP.modules.decoder.crf import allowed_transitions class CWSBiLSTMCRF(BaseModel): def __init__(self, vocab_num, embed_dim=100, bigram_vocab_num=None, bigram_embed_dim=100, num_bigram_per_char=None, @@ -183,7 +183,7 @@ class CWSBiLSTMCRF(BaseModel): masks = seq_lens_to_mask(seq_lens) feats = self.encoder_model(chars, bigrams, seq_lens) feats = self.decoder_model(feats) - probs = self.crf.viterbi_decode(feats, masks, get_score=False) + paths, _ = self.crf.viterbi_decode(feats, masks) - return {'pred': probs, 'seq_lens':seq_lens} + return {'pred': paths, 'seq_lens':seq_lens} diff --git a/reproduction/Chinese_word_segmentation/models/cws_transformer.py b/reproduction/Chinese_word_segmentation/models/cws_transformer.py new file mode 100644 index 00000000..e8ae5ecc --- /dev/null +++ b/reproduction/Chinese_word_segmentation/models/cws_transformer.py @@ -0,0 +1,199 @@ + + + +""" +使用transformer作为分词的encoder端 + +""" + +from torch import nn +import torch +# from fastNLP.modules.encoder.transformer import TransformerEncoder +from reproduction.Chinese_word_segmentation.models.transformer import TransformerEncoder +from fastNLP.modules.decoder.crf import ConditionalRandomField,seq_len_to_byte_mask +from fastNLP.modules.decoder.crf import allowed_transitions + +class TransformerCWS(nn.Module): + def __init__(self, vocab_num, embed_dim=100, bigram_vocab_num=None, bigram_embed_dim=100, num_bigram_per_char=None, + hidden_size=200, embed_drop_p=0.3, num_layers=1, num_heads=8, tag_size=4): + super().__init__() + + self.embedding = nn.Embedding(vocab_num, embed_dim) + input_size = embed_dim + if bigram_vocab_num: + self.bigram_embedding = nn.Embedding(bigram_vocab_num, bigram_embed_dim) + input_size += num_bigram_per_char*bigram_embed_dim + + self.drop = nn.Dropout(embed_drop_p, inplace=True) + + self.fc1 = nn.Linear(input_size, hidden_size) + + # value_size = hidden_size//num_heads + # self.transformer = TransformerEncoder(num_layers, model_size=hidden_size, inner_size=hidden_size, + # key_size=value_size, + # value_size=value_size, num_head=num_heads) + self.transformer = TransformerEncoder(num_layers=num_layers, model_size=hidden_size, num_heads=num_heads, + hidden_size=hidden_size) + self.fc2 = nn.Linear(hidden_size, tag_size) + + allowed_trans = allowed_transitions({0:'b', 1:'m', 2:'e', 3:'s'}, encoding_type='bmes') + self.crf = ConditionalRandomField(num_tags=tag_size, include_start_end_trans=False, + allowed_transitions=allowed_trans) + + def forward(self, chars, target, seq_lens, bigrams=None): + masks = seq_len_to_byte_mask(seq_lens) + x = self.embedding(chars) + batch_size = x.size(0) + length = x.size(1) + if hasattr(self, 'bigram_embedding'): + bigrams = self.bigram_embedding(bigrams) # batch_size x seq_lens x per_char x embed_size + x = torch.cat([x, bigrams.view(batch_size, length, -1)], dim=-1) + self.drop(x) + x = self.fc1(x) + feats = self.transformer(x, masks) + feats = self.fc2(feats) + losses = self.crf(feats, target, masks.float()) + + pred_dict = {} + pred_dict['seq_lens'] = seq_lens + pred_dict['loss'] = torch.mean(losses) + + return pred_dict + + def predict(self, chars, seq_lens, bigrams=None): + masks = seq_len_to_byte_mask(seq_lens) + + x = self.embedding(chars) + batch_size = x.size(0) + length = x.size(1) + if hasattr(self, 'bigram_embedding'): + bigrams = self.bigram_embedding(bigrams) # batch_size x seq_lens x per_char x embed_size + x = torch.cat([x, bigrams.view(batch_size, length, -1)], dim=-1) + self.drop(x) + x = self.fc1(x) + feats = self.transformer(x, masks) + feats = self.fc2(feats) + + probs = self.crf.viterbi_decode(feats, masks, get_score=False) + + return {'pred': probs, 'seq_lens':seq_lens} + + +from reproduction.Chinese_word_segmentation.models.dilated_transformer import TransformerDilateEncoder + +class TransformerDilatedCWS(nn.Module): + def __init__(self, vocab_num, embed_dim=100, bigram_vocab_num=None, bigram_embed_dim=100, num_bigram_per_char=None, + embed_drop_p=0.3, hidden_size=200, kernel_size=3, dilate='none', + num_layers=1, num_heads=8, tag_size=4, + relative_pos_embed_dim=0): + super().__init__() + + self.embedding = nn.Embedding(vocab_num, embed_dim) + input_size = embed_dim + if bigram_vocab_num: + self.bigram_embedding = nn.Embedding(bigram_vocab_num, bigram_embed_dim) + input_size += num_bigram_per_char*bigram_embed_dim + + self.drop = nn.Dropout(embed_drop_p, inplace=True) + + self.fc1 = nn.Linear(input_size, hidden_size) + + # value_size = hidden_size//num_heads + # self.transformer = TransformerEncoder(num_layers, model_size=hidden_size, inner_size=hidden_size, + # key_size=value_size, + # value_size=value_size, num_head=num_heads) + self.transformer = TransformerDilateEncoder(num_layers=num_layers, model_size=hidden_size, num_heads=num_heads, + hidden_size=hidden_size, kernel_size=kernel_size, dilate=dilate, + relative_pos_embed_dim=relative_pos_embed_dim) + self.fc2 = nn.Linear(hidden_size, tag_size) + + allowed_trans = allowed_transitions({0:'b', 1:'m', 2:'e', 3:'s'}, encoding_type='bmes') + self.crf = ConditionalRandomField(num_tags=tag_size, include_start_end_trans=False, + allowed_transitions=allowed_trans) + + def forward(self, chars, target, seq_lens, bigrams=None): + masks = seq_len_to_byte_mask(seq_lens) + x = self.embedding(chars) + batch_size = x.size(0) + length = x.size(1) + if hasattr(self, 'bigram_embedding'): + bigrams = self.bigram_embedding(bigrams) # batch_size x seq_lens x per_char x embed_size + x = torch.cat([x, bigrams.view(batch_size, length, -1)], dim=-1) + self.drop(x) + x = self.fc1(x) + feats = self.transformer(x, masks) + feats = self.fc2(feats) + losses = self.crf(feats, target, masks.float()) + + pred_dict = {} + pred_dict['seq_lens'] = seq_lens + pred_dict['loss'] = torch.mean(losses) + + return pred_dict + + def predict(self, chars, seq_lens, bigrams=None): + masks = seq_len_to_byte_mask(seq_lens) + + x = self.embedding(chars) + batch_size = x.size(0) + length = x.size(1) + if hasattr(self, 'bigram_embedding'): + bigrams = self.bigram_embedding(bigrams) # batch_size x seq_lens x per_char x embed_size + x = torch.cat([x, bigrams.view(batch_size, length, -1)], dim=-1) + self.drop(x) + x = self.fc1(x) + feats = self.transformer(x, masks) + feats = self.fc2(feats) + + paths, _ = self.crf.viterbi_decode(feats, masks) + + return {'pred': paths, 'seq_lens':seq_lens} + + + +class NoamOpt(torch.optim.Optimizer): + "Optim wrapper that implements rate." + + def __init__(self, model_size, factor, warmup, optimizer): + super().__init__([torch.nn.Parameter(torch.ones(1))], {}) + + self.optimizer = optimizer + self._step = 0 + self.warmup = warmup + self.factor = factor + self.model_size = model_size + self._rate = 0 + + def step(self, **kwargs): + "Update parameters and rate" + self._step += 1 + rate = self.rate() + for p in self.optimizer.param_groups: + p['lr'] = rate + self._rate = rate + self.optimizer.step() + + def rate(self, step=None): + "Implement `lrate` above" + if step is None: + step = self._step + return self.factor * \ + (self.model_size ** (-0.5) * + min(step ** (-0.5), step * self.warmup ** (-1.5))) + +def TransformerCWS_test(): + transformer = TransformerCWS(10, embed_dim=100, bigram_vocab_num=10, bigram_embed_dim=100, num_bigram_per_char=8, + hidden_size=200, embed_drop_p=0.3, num_layers=1, num_heads=8, tag_size=4) + chars = torch.randint(10, size=(4, 7)).long() + bigrams = torch.randint(10, size=(4, 56)).long() + seq_lens = torch.ones(4).long()*7 + target = torch.randint(4, size=(4, 7)) + + print(transformer(chars, target, seq_lens, bigrams)) + + optimizer = torch.optim.Adam(transformer.parameters()) + + opt = NoamOpt(10 ,1, 400, optimizer) + +if __name__ == '__main__': + TransformerCWS_test() diff --git a/reproduction/Chinese_word_segmentation/process/__init__.py b/reproduction/Chinese_word_segmentation/process/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/reproduction/chinese_word_segment/process/cws_processor.py b/reproduction/Chinese_word_segmentation/process/cws_processor.py similarity index 75% rename from reproduction/chinese_word_segment/process/cws_processor.py rename to reproduction/Chinese_word_segmentation/process/cws_processor.py index 9e57d35a..614d9ef5 100644 --- a/reproduction/chinese_word_segment/process/cws_processor.py +++ b/reproduction/Chinese_word_segmentation/process/cws_processor.py @@ -4,7 +4,7 @@ import re from fastNLP.api.processor import Processor from fastNLP.core.dataset import DataSet from fastNLP.core.vocabulary import Vocabulary -from reproduction.chinese_word_segment.process.span_converter import SpanConverter +from reproduction.Chinese_word_segmentation.process.span_converter import SpanConverter _SPECIAL_TAG_PATTERN = '<[a-zA-Z]+>' @@ -226,109 +226,6 @@ class Pre2Post2BigramProcessor(BigramProcessor): return bigrams -# 这里需要建立vocabulary了，但是遇到了以下的问题 -# (1) 如果使用Processor的方式的话，但是在这种情况返回的不是dataset。所以建立vocabulary的工作用另外的方式实现，不借用 -# Processor了 -# TODO 如何将建立vocab和index这两步统一了？ - -class VocabIndexerProcessor(Processor): - """ - 根据DataSet创建Vocabulary，并将其用数字index。新生成的index的field会被放在new_added_filed_name, 如果没有提供 - new_added_field_name, 则覆盖原有的field_name. - - """ - def __init__(self, field_name, new_added_filed_name=None, min_freq=1, max_size=None, - verbose=0, is_input=True): - """ - - :param field_name: 从哪个field_name创建词表，以及对哪个field_name进行index操作 - :param new_added_filed_name: index时，生成的index field的名称，如果不传入，则覆盖field_name. - :param min_freq: 创建的Vocabulary允许的单词最少出现次数. - :param max_size: 创建的Vocabulary允许的最大的单词数量 - :param verbose: 0, 不输出任何信息；1，输出信息 - :param bool is_input: - """ - super(VocabIndexerProcessor, self).__init__(field_name, new_added_filed_name) - self.min_freq = min_freq - self.max_size = max_size - - self.verbose =verbose - self.is_input = is_input - - def construct_vocab(self, *datasets): - """ - 使用传入的DataSet创建vocabulary - - :param datasets: DataSet类型的数据，用于构建vocabulary - :return: - """ - self.vocab = Vocabulary(min_freq=self.min_freq, max_size=self.max_size) - for dataset in datasets: - assert isinstance(dataset, DataSet), "Only Dataset class is allowed, not {}.".format(type(dataset)) - dataset.apply(lambda ins: self.vocab.update(ins[self.field_name])) - self.vocab.build_vocab() - if self.verbose: - print("Vocabulary Constructed, has {} items.".format(len(self.vocab))) - - def process(self, *datasets, only_index_dataset=None): - """ - 若还未建立Vocabulary，则使用dataset中的DataSet建立vocabulary；若已经有了vocabulary则使用已有的vocabulary。得到vocabulary - 后，则会index datasets与only_index_dataset。 - - :param datasets: DataSet类型的数据 - :param only_index_dataset: DataSet, or list of DataSet. 该参数中的内容只会被用于index，不会被用于生成vocabulary。 - :return: - """ - if len(datasets)==0 and not hasattr(self,'vocab'): - raise RuntimeError("You have to construct vocabulary first. Or you have to pass datasets to construct it.") - if not hasattr(self, 'vocab'): - self.construct_vocab(*datasets) - else: - if self.verbose: - print("Using constructed vocabulary with {} items.".format(len(self.vocab))) - to_index_datasets = [] - if len(datasets)!=0: - for dataset in datasets: - assert isinstance(dataset, DataSet), "Only DataSet class is allowed, not {}.".format(type(dataset)) - to_index_datasets.append(dataset) - - if not (only_index_dataset is None): - if isinstance(only_index_dataset, list): - for dataset in only_index_dataset: - assert isinstance(dataset, DataSet), "Only DataSet class is allowed, not {}.".format(type(dataset)) - to_index_datasets.append(dataset) - elif isinstance(only_index_dataset, DataSet): - to_index_datasets.append(only_index_dataset) - else: - raise TypeError('Only DataSet or list of DataSet is allowed, not {}.'.format(type(only_index_dataset))) - - for dataset in to_index_datasets: - assert isinstance(dataset, DataSet), "Only DataSet class is allowed, not {}.".format(type(dataset)) - dataset.apply(lambda ins: [self.vocab.to_index(token) for token in ins[self.field_name]], - new_field_name=self.new_added_field_name, is_input=self.is_input) - # 只返回一个，infer时为了跟其他processor保持一致 - if len(to_index_datasets) == 1: - return to_index_datasets[0] - - def set_vocab(self, vocab): - assert isinstance(vocab, Vocabulary), "Only fastNLP.core.Vocabulary is allowed, not {}.".format(type(vocab)) - self.vocab = vocab - - def delete_vocab(self): - del self.vocab - - def get_vocab_size(self): - return len(self.vocab) - - def set_verbose(self, verbose): - """ - 设置processor verbose状态。 - - :param verbose: int, 0，不输出任何信息；1，输出vocab 信息。 - :return: - """ - self.verbose = verbose - class VocabProcessor(Processor): def __init__(self, field_name, min_freq=1, max_size=None): diff --git a/reproduction/chinese_word_segment/process/span_converter.py b/reproduction/Chinese_word_segmentation/process/span_converter.py similarity index 100% rename from reproduction/chinese_word_segment/process/span_converter.py rename to reproduction/Chinese_word_segmentation/process/span_converter.py diff --git a/reproduction/chinese_word_segment/utils.py b/reproduction/Chinese_word_segmentation/utils.py similarity index 100% rename from reproduction/chinese_word_segment/utils.py rename to reproduction/Chinese_word_segmentation/utils.py diff --git a/reproduction/LSTM+self_attention_sentiment_analysis/main.py b/reproduction/LSTM+self_attention_sentiment_analysis/main.py index 61ab79f4..871dc476 100644 --- a/reproduction/LSTM+self_attention_sentiment_analysis/main.py +++ b/reproduction/LSTM+self_attention_sentiment_analysis/main.py @@ -4,10 +4,10 @@ from fastNLP.core.trainer import ClassificationTrainer from fastNLP.core.utils import ClassPreprocess as Preprocess from fastNLP.io.config_io import ConfigLoader from fastNLP.io.config_io import ConfigSection -from fastNLP.io.dataset_loader import ClassDataSetLoader as Dataset_loader +from fastNLP.io.dataset_loader import DummyClassificationReader as Dataset_loader from fastNLP.models.base_model import BaseModel from fastNLP.modules.aggregator.self_attention import SelfAttention -from fastNLP.modules.decoder.MLP import MLP +from fastNLP.modules.decoder.mlp import MLP from fastNLP.modules.encoder.embedding import Embedding as Embedding from fastNLP.modules.encoder.lstm import LSTM @@ -42,7 +42,7 @@ train_data, dev_data = preprocess.run(train_data, dev_data) class SELF_ATTENTION_YELP_CLASSIFICATION(BaseModel): def __init__(self, args=None): super(SELF_ATTENTION_YELP_CLASSIFICATION,self).__init__() - self.embedding = Embedding(len(word2index) ,embeding_size , init_emb= None ) + self.embedding = Embedding((len(word2index) ,embeding_size)) self.lstm = LSTM(input_size=embeding_size, hidden_size=lstm_hidden_size, bidirectional=True) self.attention = SelfAttention(lstm_hidden_size * 2 ,dim =attention_unit ,num_vec=attention_hops) self.mlp = MLP(size_layer=[lstm_hidden_size * 2*attention_hops ,nfc ,class_num ]) diff --git a/reproduction/pos_tag_model/pos_processor.py b/reproduction/POS_tagging/pos_processor.py similarity index 100% rename from reproduction/pos_tag_model/pos_processor.py rename to reproduction/POS_tagging/pos_processor.py diff --git a/reproduction/POS_tagging/pos_reader.py b/reproduction/POS_tagging/pos_reader.py new file mode 100644 index 00000000..4ff58f4b --- /dev/null +++ b/reproduction/POS_tagging/pos_reader.py @@ -0,0 +1,29 @@ +from fastNLP.io.dataset_loader import ZhConllPOSReader + + +def cut_long_sentence(sent, max_sample_length=200): + sent_no_space = sent.replace(' ', '') + cutted_sentence = [] + if len(sent_no_space) > max_sample_length: + parts = sent.strip().split() + new_line = '' + length = 0 + for part in parts: + length += len(part) + new_line += part + ' ' + if length > max_sample_length: + new_line = new_line[:-1] + cutted_sentence.append(new_line) + length = 0 + new_line = '' + if new_line != '': + cutted_sentence.append(new_line[:-1]) + else: + cutted_sentence.append(sent) + return cutted_sentence + + +if __name__ == '__main__': + reader = ZhConllPOSReader() + d = reader.load('/home/hyan/train.conllx') + print(d) \ No newline at end of file diff --git a/reproduction/pos_tag_model/pos_tag.cfg b/reproduction/POS_tagging/pos_tag.cfg similarity index 94% rename from reproduction/pos_tag_model/pos_tag.cfg rename to reproduction/POS_tagging/pos_tag.cfg index c9ee8320..f8224234 100644 --- a/reproduction/pos_tag_model/pos_tag.cfg +++ b/reproduction/POS_tagging/pos_tag.cfg @@ -10,7 +10,7 @@ eval_sort_key = 'accuracy' [model] rnn_hidden_units = 300 -word_emb_dim = 100 +word_emb_dim = 300 dropout = 0.5 use_crf = true print_every_step = 10 diff --git a/reproduction/POS_tagging/train_pos_tag.py b/reproduction/POS_tagging/train_pos_tag.py new file mode 100644 index 00000000..ccf7aa1e --- /dev/null +++ b/reproduction/POS_tagging/train_pos_tag.py @@ -0,0 +1,163 @@ +import argparse +import os +import pickle +import sys + +import torch + +# in order to run fastNLP without installation +sys.path.append(os.path.join(os.path.dirname(__file__), '../..')) + +from fastNLP.api.pipeline import Pipeline +from fastNLP.api.processor import SeqLenProcessor, VocabIndexerProcessor, SetInputProcessor, IndexerProcessor +from fastNLP.core.metrics import SpanFPreRecMetric +from fastNLP.core.trainer import Trainer +from fastNLP.io.config_io import ConfigLoader, ConfigSection +from fastNLP.models.sequence_labeling import AdvSeqLabel +from fastNLP.io.dataset_loader import ConllxDataLoader +from fastNLP.api.processor import ModelProcessor, Index2WordProcessor + + +cfgfile = './pos_tag.cfg' +pickle_path = "save" + + +def load_tencent_embed(embed_path, word2id): + hit = 0 + with open(embed_path, "rb") as f: + embed_dict = pickle.load(f) + embedding_tensor = torch.randn(len(word2id), 200) + for key in word2id: + if key in embed_dict: + embedding_tensor[word2id[key]] = torch.Tensor(embed_dict[key]) + hit += 1 + print("vocab_size={} hit={} hit/vocab_size={}".format(len(word2id), hit, hit / len(word2id))) + return embedding_tensor + + +def train(train_data_path, dev_data_path, checkpoint=None, save=None): + # load config + train_param = ConfigSection() + model_param = ConfigSection() + ConfigLoader().load_config(cfgfile, {"train": train_param, "model": model_param}) + print("config loaded") + + # Data Loader + print("loading training set...") + dataset = ConllxDataLoader().load(train_data_path, return_dataset=True) + print("loading dev set...") + dev_data = ConllxDataLoader().load(dev_data_path, return_dataset=True) + print(dataset) + print("================= dataset ready =====================") + + dataset.rename_field("tag", "truth") + dev_data.rename_field("tag", "truth") + + vocab_proc = VocabIndexerProcessor("words", new_added_filed_name="word_seq") + tag_proc = VocabIndexerProcessor("truth", is_input=True) + seq_len_proc = SeqLenProcessor(field_name="word_seq", new_added_field_name="word_seq_origin_len", is_input=True) + set_input_proc = SetInputProcessor("word_seq", "word_seq_origin_len") + + vocab_proc(dataset) + tag_proc(dataset) + seq_len_proc(dataset) + + # index dev set + word_vocab, tag_vocab = vocab_proc.vocab, tag_proc.vocab + dev_data.apply(lambda ins: [word_vocab.to_index(w) for w in ins["words"]], new_field_name="word_seq") + dev_data.apply(lambda ins: [tag_vocab.to_index(w) for w in ins["truth"]], new_field_name="truth") + dev_data.apply(lambda ins: len(ins["word_seq"]), new_field_name="word_seq_origin_len") + + # set input & target + dataset.set_input("word_seq", "word_seq_origin_len", "truth") + dev_data.set_input("word_seq", "word_seq_origin_len", "truth") + dataset.set_target("truth", "word_seq_origin_len") + dev_data.set_target("truth", "word_seq_origin_len") + + # dataset.set_is_target(tag_ids=True) + model_param["vocab_size"] = vocab_proc.get_vocab_size() + model_param["num_classes"] = tag_proc.get_vocab_size() + print("vocab_size={} num_classes={}".format(model_param["vocab_size"], model_param["num_classes"])) + + # define a model + if checkpoint is None: + # pre_trained = load_tencent_embed("/home/zyfeng/data/char_tencent_embedding.pkl", vocab_proc.vocab.word2idx) + pre_trained = None + model = AdvSeqLabel(model_param, id2words=None, emb=pre_trained) + print(model) + else: + model = torch.load(checkpoint) + + # call trainer to train + trainer = Trainer(dataset, model, loss=None, metrics=SpanFPreRecMetric(tag_proc.vocab, pred="predict", + target="truth", + seq_lens="word_seq_origin_len"), + dev_data=dev_data, metric_key="f", + use_tqdm=True, use_cuda=True, print_every=10, n_epochs=20, save_path=save) + trainer.train(load_best_model=True) + + # save model & pipeline + model_proc = ModelProcessor(model, seq_len_field_name="word_seq_origin_len") + id2tag = Index2WordProcessor(tag_proc.vocab, "predict", "tag") + + pp = Pipeline([vocab_proc, seq_len_proc, set_input_proc, model_proc, id2tag]) + save_dict = {"pipeline": pp, "model": model, "tag_vocab": tag_proc.vocab} + torch.save(save_dict, os.path.join(save, "model_pp.pkl")) + print("pipeline saved") + + +def run_test(test_path): + test_data = ConllxDataLoader().load(test_path, return_dataset=True) + + with open("model_pp_0117.pkl", "rb") as f: + save_dict = torch.load(f) + tag_vocab = save_dict["tag_vocab"] + pipeline = save_dict["pipeline"] + index_tag = IndexerProcessor(vocab=tag_vocab, field_name="tag", new_added_field_name="truth", is_input=False) + pipeline.pipeline = [index_tag] + pipeline.pipeline + + pipeline(test_data) + test_data.set_target("truth") + prediction = test_data.field_arrays["predict"].content + truth = test_data.field_arrays["truth"].content + seq_len = test_data.field_arrays["word_seq_origin_len"].content + + # padding by hand + max_length = max([len(seq) for seq in prediction]) + for idx in range(len(prediction)): + prediction[idx] = list(prediction[idx]) + ([0] * (max_length - len(prediction[idx]))) + truth[idx] = list(truth[idx]) + ([0] * (max_length - len(truth[idx]))) + evaluator = SpanFPreRecMetric(tag_vocab=tag_vocab, pred="predict", target="truth", + seq_lens="word_seq_origin_len") + evaluator({"predict": torch.Tensor(prediction), "word_seq_origin_len": torch.Tensor(seq_len)}, + {"truth": torch.Tensor(truth)}) + test_result = evaluator.get_metric() + f1 = round(test_result['f'] * 100, 2) + pre = round(test_result['pre'] * 100, 2) + rec = round(test_result['rec'] * 100, 2) + + return {"F1": f1, "precision": pre, "recall": rec} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--train", type=str, help="training conll file", default="/home/zyfeng/data/sample.conllx") + parser.add_argument("--dev", type=str, help="dev conll file", default="/home/zyfeng/data/sample.conllx") + parser.add_argument("--test", type=str, help="test conll file", default=None) + parser.add_argument("--save", type=str, help="path to save", default=None) + + parser.add_argument("-c", "--restart", action="store_true", help="whether to continue training") + parser.add_argument("-cp", "--checkpoint", type=str, help="checkpoint of the trained model") + args = parser.parse_args() + + if args.test is not None: + print(run_test(args.test)) + else: + if args.restart is True: + # 继续训练 python train_pos_tag.py -c -cp ./save/best_model.pkl + if args.checkpoint is None: + raise RuntimeError("Please provide the checkpoint. -cp ") + train(args.train, args.dev, args.checkpoint, save=args.save) + else: + # 一次训练 python train_pos_tag.py + train(args.train, args.dev, save=args.save) diff --git a/reproduction/pos_tag_model/utils.py b/reproduction/POS_tagging/utils.py similarity index 100% rename from reproduction/pos_tag_model/utils.py rename to reproduction/POS_tagging/utils.py diff --git a/reproduction/README.md b/reproduction/README.md new file mode 100644 index 00000000..8d14d36d --- /dev/null +++ b/reproduction/README.md @@ -0,0 +1,44 @@ +# 模型复现 +这里复现了在fastNLP中实现的模型，旨在达到与论文中相符的性能。 + +复现的模型有: +- Star-Transformer +- ... + + +## Star-Transformer +[reference](https://arxiv.org/abs/1902.09113) +### Performance (still in progress) +|任务| 数据集 | SOTA | 模型表现 | +|------|------| ------| ------| +|Pos Tagging|CTB 9.0|-|ACC 92.31| +|Pos Tagging|CONLL 2012|-|ACC 96.51| +|Named Entity Recognition|CONLL 2012|-|F1 85.66| +|Text Classification|SST|-|49.18| +|Natural Language Inference|SNLI|-|83.76| + +### Usage +``` python +# for sequence labeling(ner, pos tagging, etc) +from fastNLP.models.star_transformer import STSeqLabel +model = STSeqLabel( + vocab_size=10000, num_cls=50, + emb_dim=300) + + +# for sequence classification +from fastNLP.models.star_transformer import STSeqCls +model = STSeqCls( + vocab_size=10000, num_cls=50, + emb_dim=300) + + +# for natural language inference +from fastNLP.models.star_transformer import STNLICls +model = STNLICls( + vocab_size=10000, num_cls=50, + emb_dim=300) + +``` + +## ... diff --git a/reproduction/Star_transformer/datasets.py b/reproduction/Star_transformer/datasets.py new file mode 100644 index 00000000..a9257fd4 --- /dev/null +++ b/reproduction/Star_transformer/datasets.py @@ -0,0 +1,157 @@ +import torch +import json +import os +from fastNLP import Vocabulary +from fastNLP.io.dataset_loader import ConllLoader, SSTLoader, SNLILoader +from fastNLP.core import Const as C +import numpy as np + +MAX_LEN = 128 + +def update_v(vocab, data, field): + data.apply(lambda x: vocab.add_word_lst(x[field]), new_field_name=None) + + +def to_index(vocab, data, field, name): + def func(x): + try: + return [vocab.to_index(w) for w in x[field]] + except ValueError: + return [vocab.padding_idx for _ in x[field]] + data.apply(func, new_field_name=name) + + +def load_seqtag(path, files, indexs): + word_h, tag_h = 'words', 'tags' + loader = ConllLoader(headers=[word_h, tag_h], indexes=indexs) + ds_list = [] + for fn in files: + ds_list.append(loader.load(os.path.join(path, fn))) + word_v = Vocabulary(min_freq=2) + tag_v = Vocabulary(unknown=None) + update_v(word_v, ds_list[0], word_h) + update_v(tag_v, ds_list[0], tag_h) + + def process_data(ds): + to_index(word_v, ds, word_h, C.INPUT) + to_index(tag_v, ds, tag_h, C.TARGET) + ds.apply(lambda x: x[C.INPUT][:MAX_LEN], new_field_name=C.INPUT) + ds.apply(lambda x: x[C.TARGET][:MAX_LEN], new_field_name=C.TARGET) + ds.apply(lambda x: len(x[word_h]), new_field_name=C.INPUT_LEN) + ds.set_input(C.INPUT, C.INPUT_LEN) + ds.set_target(C.TARGET, C.INPUT_LEN) + for i in range(len(ds_list)): + process_data(ds_list[i]) + return ds_list, word_v, tag_v + + +def load_sst(path, files): + loaders = [SSTLoader(subtree=sub, fine_grained=True) + for sub in [True, False, False]] + ds_list = [loader.load(os.path.join(path, fn)) + for fn, loader in zip(files, loaders)] + word_v = Vocabulary(min_freq=2) + tag_v = Vocabulary(unknown=None, padding=None) + for ds in ds_list: + ds.apply(lambda x: [w.lower() + for w in x['words']], new_field_name='words') + ds_list[0].drop(lambda x: len(x['words']) < 3) + update_v(word_v, ds_list[0], 'words') + ds_list[0].apply(lambda x: tag_v.add_word( + x['target']), new_field_name=None) + + def process_data(ds): + to_index(word_v, ds, 'words', C.INPUT) + ds.apply(lambda x: tag_v.to_index(x['target']), new_field_name=C.TARGET) + ds.apply(lambda x: x[C.INPUT][:MAX_LEN], new_field_name=C.INPUT) + ds.apply(lambda x: len(x['words']), new_field_name=C.INPUT_LEN) + ds.set_input(C.INPUT, C.INPUT_LEN) + ds.set_target(C.TARGET) + for i in range(len(ds_list)): + process_data(ds_list[i]) + return ds_list, word_v, tag_v + + +def load_snli(path, files): + loader = SNLILoader() + ds_list = [loader.load(os.path.join(path, f)) for f in files] + word_v = Vocabulary(min_freq=2) + tag_v = Vocabulary(unknown=None, padding=None) + for ds in ds_list: + ds.apply(lambda x: [w.lower() + for w in x['words1']], new_field_name='words1') + ds.apply(lambda x: [w.lower() + for w in x['words2']], new_field_name='words2') + update_v(word_v, ds_list[0], 'words1') + update_v(word_v, ds_list[0], 'words2') + ds_list[0].apply(lambda x: tag_v.add_word( + x['target']), new_field_name=None) + + def process_data(ds): + to_index(word_v, ds, 'words1', C.INPUTS(0)) + to_index(word_v, ds, 'words2', C.INPUTS(1)) + ds.apply(lambda x: tag_v.to_index(x['target']), new_field_name=C.TARGET) + ds.apply(lambda x: x[C.INPUTS(0)][:MAX_LEN], new_field_name=C.INPUTS(0)) + ds.apply(lambda x: x[C.INPUTS(1)][:MAX_LEN], new_field_name=C.INPUTS(1)) + ds.apply(lambda x: len(x[C.INPUTS(0)]), new_field_name=C.INPUT_LENS(0)) + ds.apply(lambda x: len(x[C.INPUTS(1)]), new_field_name=C.INPUT_LENS(1)) + ds.set_input(C.INPUTS(0), C.INPUTS(1), C.INPUT_LENS(0), C.INPUT_LENS(1)) + ds.set_target(C.TARGET) + for i in range(len(ds_list)): + process_data(ds_list[i]) + return ds_list, word_v, tag_v + + +class EmbedLoader: + @staticmethod + def parse_glove_line(line): + line = line.split() + if len(line) <= 2: + raise RuntimeError( + "something goes wrong in parsing glove embedding") + return line[0], line[1:] + + @staticmethod + def str_list_2_vec(line): + return torch.Tensor(list(map(float, line))) + + @staticmethod + def fast_load_embedding(emb_dim, emb_file, vocab): + """Fast load the pre-trained embedding and combine with the given dictionary. + This loading method uses line-by-line operation. + + :param int emb_dim: the dimension of the embedding. Should be the same as pre-trained embedding. + :param str emb_file: the pre-trained embedding file path. + :param Vocabulary vocab: a mapping from word to index, can be provided by user or built from pre-trained embedding + :return embedding_matrix: numpy.ndarray + + """ + if vocab is None: + raise RuntimeError("You must provide a vocabulary.") + embedding_matrix = np.zeros( + shape=(len(vocab), emb_dim), dtype=np.float32) + hit_flags = np.zeros(shape=(len(vocab),), dtype=int) + with open(emb_file, "r", encoding="utf-8") as f: + startline = f.readline() + if len(startline.split()) > 2: + f.seek(0) + for line in f: + word, vector = EmbedLoader.parse_glove_line(line) + try: + if word in vocab: + vector = EmbedLoader.str_list_2_vec(vector) + if emb_dim != vector.size(0): + continue + embedding_matrix[vocab[word]] = vector + hit_flags[vocab[word]] = 1 + except Exception: + continue + + if np.sum(hit_flags) < len(vocab): + # some words from vocab are missing in pre-trained embedding + # we normally sample each dimension + vocab_embed = embedding_matrix[np.where(hit_flags)] + sampled_vectors = np.random.normal(vocab_embed.mean(axis=0), vocab_embed.std(axis=0), + size=(len(vocab) - np.sum(hit_flags), emb_dim)) + embedding_matrix[np.where(1 - hit_flags)] = sampled_vectors + return embedding_matrix diff --git a/reproduction/Star_transformer/modules.py b/reproduction/Star_transformer/modules.py new file mode 100644 index 00000000..61a61d25 --- /dev/null +++ b/reproduction/Star_transformer/modules.py @@ -0,0 +1,56 @@ +import torch +import torch.nn as nn +import torch.nn.functional as F +import numpy as np +from fastNLP.core.losses import LossBase + + +reduce_func = { + 'none': lambda x, mask: x*mask, + 'sum': lambda x, mask: (x*mask).sum(), + 'mean': lambda x, mask: (x*mask).sum() / mask.sum(), +} + + +class LabelSmoothCrossEntropy(nn.Module): + def __init__(self, smoothing=0.1, ignore_index=-100, reduction='mean'): + global reduce_func + super().__init__() + if smoothing < 0 or smoothing > 1: + raise ValueError('invalid smoothing value: {}'.format(smoothing)) + self.smoothing = smoothing + self.ignore_index = ignore_index + if reduction not in reduce_func: + raise ValueError('invalid reduce type: {}'.format(reduction)) + self.reduce_func = reduce_func[reduction] + + def forward(self, input, target): + input = F.log_softmax(input, dim=1) # [N, C, ...] + smooth_val = self.smoothing / input.size(1) # [N, C, ...] + target_logit = input.new_full(input.size(), fill_value=smooth_val) + target_logit.scatter_(1, target[:, None], 1 - self.smoothing) + result = -(target_logit * input).sum(1) # [N, ...] + mask = (target != self.ignore_index).float() + return self.reduce_func(result, mask) + + +class SmoothCE(LossBase): + def __init__(self, pred=None, target=None, **kwargs): + super().__init__() + self.loss_fn = LabelSmoothCrossEntropy(**kwargs) + self._init_param_map(pred=pred, target=target) + + def get_loss(self, pred, target): + return self.loss_fn(pred, target) + + +if __name__ == '__main__': + loss_fn = nn.CrossEntropyLoss(ignore_index=0) + sm_loss_fn = LabelSmoothCrossEntropy(smoothing=0, ignore_index=0) + predict = torch.tensor([[0, 0.2, 0.7, 0.1, 0], + [0, 0.9, 0.2, 0.1, 0], + [1, 0.2, 0.7, 0.1, 0]]) + target = torch.tensor([2, 1, 0]) + loss = loss_fn(predict, target) + sm_loss = sm_loss_fn(predict, target) + print(loss, sm_loss) diff --git a/reproduction/Star_transformer/run.sh b/reproduction/Star_transformer/run.sh new file mode 100644 index 00000000..0972c662 --- /dev/null +++ b/reproduction/Star_transformer/run.sh @@ -0,0 +1,5 @@ +#python -u train.py --task pos --ds conll --mode train --gpu 1 --lr 3e-4 --w_decay 2e-5 --lr_decay .95 --drop 0.3 --ep 25 --bsz 64 > conll_pos102.log 2>&1 & +#python -u train.py --task pos --ds ctb --mode train --gpu 1 --lr 3e-4 --w_decay 2e-5 --lr_decay .95 --drop 0.3 --ep 25 --bsz 64 > ctb_pos101.log 2>&1 & +#python -u train.py --task cls --ds sst --mode train --gpu 2 --lr 1e-4 --w_decay 1e-5 --lr_decay 0.9 --drop 0.5 --ep 50 --bsz 128 > sst_cls201.log & +#python -u train.py --task nli --ds snli --mode train --gpu 1 --lr 1e-4 --w_decay 1e-5 --lr_decay 0.9 --drop 0.4 --ep 120 --bsz 128 > snli_nli201.log & +python -u train.py --task ner --ds conll --mode train --gpu 0 --lr 1e-4 --w_decay 1e-5 --lr_decay 0.9 --drop 0.4 --ep 120 --bsz 64 > conll_ner201.log & diff --git a/reproduction/Star_transformer/train.py b/reproduction/Star_transformer/train.py new file mode 100644 index 00000000..dee85c38 --- /dev/null +++ b/reproduction/Star_transformer/train.py @@ -0,0 +1,214 @@ +from util import get_argparser, set_gpu, set_rng_seeds, add_model_args +from datasets import load_seqtag, load_sst, load_snli, EmbedLoader, MAX_LEN +import torch.nn as nn +import torch +import numpy as np +import fastNLP as FN +from fastNLP.models.star_transformer import STSeqLabel, STSeqCls, STNLICls +from fastNLP.core.const import Const as C +import sys +sys.path.append('/remote-home/yfshao/workdir/dev_fastnlp/') + + +g_model_select = { + 'pos': STSeqLabel, + 'ner': STSeqLabel, + 'cls': STSeqCls, + 'nli': STNLICls, +} + +g_emb_file_path = {'en': '/remote-home/yfshao/workdir/datasets/word_vector/glove.840B.300d.txt', + 'zh': '/remote-home/yfshao/workdir/datasets/word_vector/cc.zh.300.vec'} + +g_args = None +g_model_cfg = None + + +def get_ptb_pos(): + pos_dir = '/remote-home/yfshao/workdir/datasets/pos' + pos_files = ['train.pos', 'dev.pos', 'test.pos', ] + return load_seqtag(pos_dir, pos_files, [0, 1]) + + +def get_ctb_pos(): + ctb_dir = '/remote-home/yfshao/workdir/datasets/ctb9_hy' + files = ['train.conllx', 'dev.conllx', 'test.conllx'] + return load_seqtag(ctb_dir, files, [1, 4]) + + +def get_conll2012_pos(): + path = '/remote-home/yfshao/workdir/datasets/ontonotes/pos' + files = ['ontonotes-conll.train', + 'ontonotes-conll.dev', + 'ontonotes-conll.conll-2012-test'] + return load_seqtag(path, files, [0, 1]) + + +def get_conll2012_ner(): + path = '/remote-home/yfshao/workdir/datasets/ontonotes/ner' + files = ['bieso-ontonotes-conll-ner.train', + 'bieso-ontonotes-conll-ner.dev', + 'bieso-ontonotes-conll-ner.conll-2012-test'] + return load_seqtag(path, files, [0, 1]) + + +def get_sst(): + path = '/remote-home/yfshao/workdir/datasets/SST' + files = ['train.txt', 'dev.txt', 'test.txt'] + return load_sst(path, files) + + +def get_snli(): + path = '/remote-home/yfshao/workdir/datasets/nli-data/snli_1.0' + files = ['snli_1.0_train.jsonl', + 'snli_1.0_dev.jsonl', 'snli_1.0_test.jsonl'] + return load_snli(path, files) + + +g_datasets = { + 'ptb-pos': get_ptb_pos, + 'ctb-pos': get_ctb_pos, + 'conll-pos': get_conll2012_pos, + 'conll-ner': get_conll2012_ner, + 'sst-cls': get_sst, + 'snli-nli': get_snli, +} + + +def load_pretrain_emb(word_v, lang='en'): + print('loading pre-train embeddings') + emb = EmbedLoader.fast_load_embedding(300, g_emb_file_path[lang], word_v) + emb /= np.linalg.norm(emb, axis=1, keepdims=True) + emb = torch.tensor(emb, dtype=torch.float32) + print('embedding mean: {:.6}, std: {:.6}'.format(emb.mean(), emb.std())) + emb[word_v.padding_idx].fill_(0) + return emb + + +class MyCallback(FN.core.callback.Callback): + def on_train_begin(self): + super(MyCallback, self).on_train_begin() + self.init_lrs = [pg['lr'] for pg in self.optimizer.param_groups] + + def on_backward_end(self): + nn.utils.clip_grad.clip_grad_norm_(self.model.parameters(), 5.0) + + def on_step_end(self): + warm_steps = 6000 + # learning rate warm-up & decay + if self.step <= warm_steps: + for lr, pg in zip(self.init_lrs, self.optimizer.param_groups): + pg['lr'] = lr * (self.step / float(warm_steps)) + + elif self.step % 3000 == 0: + for pg in self.optimizer.param_groups: + cur_lr = pg['lr'] + pg['lr'] = max(1e-5, cur_lr*g_args.lr_decay) + + + +def train(): + seed = set_rng_seeds(1234) + print('RNG SEED {}'.format(seed)) + print('loading data') + ds_list, word_v, tag_v = g_datasets['{}-{}'.format( + g_args.ds, g_args.task)]() + print(ds_list[0][:2]) + embed = load_pretrain_emb(word_v, lang='zh' if g_args.ds == 'ctb' else 'en') + g_model_cfg['num_cls'] = len(tag_v) + print(g_model_cfg) + g_model_cfg['init_embed'] = embed + model = g_model_select[g_args.task.lower()](**g_model_cfg) + + def init_model(model): + for p in model.parameters(): + if p.size(0) != len(word_v): + nn.init.normal_(p, 0.0, 0.05) + init_model(model) + train_data = ds_list[0] + dev_data = ds_list[2] + test_data = ds_list[1] + print(tag_v.word2idx) + + if g_args.task in ['pos', 'ner']: + padding_idx = tag_v.padding_idx + else: + padding_idx = -100 + print('padding_idx ', padding_idx) + loss = FN.CrossEntropyLoss(padding_idx=padding_idx) + metrics = { + 'pos': (None, FN.AccuracyMetric()), + 'ner': ('f', FN.core.metrics.SpanFPreRecMetric( + tag_vocab=tag_v, encoding_type='bmeso', ignore_labels=[''], )), + 'cls': (None, FN.AccuracyMetric()), + 'nli': (None, FN.AccuracyMetric()), + } + metric_key, metric = metrics[g_args.task] + device = 'cuda' if torch.cuda.is_available() else 'cpu' + ex_param = [x for x in model.parameters( + ) if x.requires_grad and x.size(0) != len(word_v)] + optim_cfg = [{'params': model.enc.embedding.parameters(), 'lr': g_args.lr*0.1}, + {'params': ex_param, 'lr': g_args.lr, 'weight_decay': g_args.w_decay}, ] + trainer = FN.Trainer(model=model, train_data=train_data, dev_data=dev_data, + loss=loss, metrics=metric, metric_key=metric_key, + optimizer=torch.optim.Adam(optim_cfg), + n_epochs=g_args.ep, batch_size=g_args.bsz, print_every=10, validate_every=3000, + device=device, + use_tqdm=False, prefetch=False, + save_path=g_args.log, + callbacks=[MyCallback()]) + + trainer.train() + tester = FN.Tester(data=test_data, model=model, metrics=metric, + batch_size=128, device=device) + tester.test() + + +def test(): + pass + + +def infer(): + pass + + +run_select = { + 'train': train, + 'test': test, + 'infer': infer, +} + + +def main(): + global g_args, g_model_cfg + import signal + + def signal_handler(signal, frame): + raise KeyboardInterrupt + signal.signal(signal.SIGINT, signal_handler) + signal.signal(signal.SIGTERM, signal_handler) + parser = get_argparser() + parser.add_argument('--task', choices=['pos', 'ner', 'cls', 'nli']) + parser.add_argument('--mode', choices=['train', 'test', 'infer']) + parser.add_argument('--ds', type=str) + add_model_args(parser) + g_args = parser.parse_args() + print(g_args.__dict__) + set_gpu(g_args.gpu) + g_model_cfg = { + 'init_embed': (None, 300), + 'num_cls': None, + 'hidden_size': g_args.hidden, + 'num_layers': 4, + 'num_head': g_args.nhead, + 'head_dim': g_args.hdim, + 'max_len': MAX_LEN, + 'cls_hidden_size': 600, + 'emb_dropout': 0.3, + 'dropout': g_args.drop, + } + run_select[g_args.mode.lower()]() + + +if __name__ == '__main__': + main() diff --git a/reproduction/Star_transformer/util.py b/reproduction/Star_transformer/util.py new file mode 100644 index 00000000..ecd1e18d --- /dev/null +++ b/reproduction/Star_transformer/util.py @@ -0,0 +1,112 @@ +import fastNLP as FN +import argparse +import os +import random +import numpy +import torch + + +def get_argparser(): + parser = argparse.ArgumentParser() + parser.add_argument('--lr', type=float, required=True) + parser.add_argument('--w_decay', type=float, required=True) + parser.add_argument('--lr_decay', type=float, required=True) + parser.add_argument('--bsz', type=int, required=True) + parser.add_argument('--ep', type=int, required=True) + parser.add_argument('--drop', type=float, required=True) + parser.add_argument('--gpu', type=str, required=True) + parser.add_argument('--log', type=str, default=None) + return parser + + +def add_model_args(parser): + parser.add_argument('--nhead', type=int, default=6) + parser.add_argument('--hdim', type=int, default=50) + parser.add_argument('--hidden', type=int, default=300) + return parser + + +def set_gpu(gpu_str): + os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" + os.environ['CUDA_VISIBLE_DEVICES'] = gpu_str + + +def set_rng_seeds(seed=None): + if seed is None: + seed = numpy.random.randint(0, 65536) + random.seed(seed) + numpy.random.seed(seed) + torch.random.manual_seed(seed) + torch.cuda.manual_seed_all(seed) + # print('RNG_SEED {}'.format(seed)) + return seed + + +class TensorboardCallback(FN.Callback): + """ + 接受以下一个或多个字符串作为参数： + - "model" + - "loss" + - "metric" + """ + + def __init__(self, *options): + super(TensorboardCallback, self).__init__() + args = {"model", "loss", "metric"} + for opt in options: + if opt not in args: + raise ValueError( + "Unrecognized argument {}. Expect one of {}".format(opt, args)) + self.options = options + self._summary_writer = None + self.graph_added = False + + def on_train_begin(self): + save_dir = self.trainer.save_path + if save_dir is None: + path = os.path.join( + "./", 'tensorboard_logs_{}'.format(self.trainer.start_time)) + else: + path = os.path.join( + save_dir, 'tensorboard_logs_{}'.format(self.trainer.start_time)) + self._summary_writer = SummaryWriter(path) + + def on_batch_begin(self, batch_x, batch_y, indices): + if "model" in self.options and self.graph_added is False: + # tesorboardX 这里有大bug，暂时没法画模型图 + # from fastNLP.core.utils import _build_args + # inputs = _build_args(self.trainer.model, **batch_x) + # args = tuple([value for value in inputs.values()]) + # args = args[0] if len(args) == 1 else args + # self._summary_writer.add_graph(self.trainer.model, torch.zeros(32, 2)) + self.graph_added = True + + def on_backward_begin(self, loss): + if "loss" in self.options: + self._summary_writer.add_scalar( + "loss", loss.item(), global_step=self.trainer.step) + + if "model" in self.options: + for name, param in self.trainer.model.named_parameters(): + if param.requires_grad: + self._summary_writer.add_scalar( + name + "_mean", param.mean(), global_step=self.trainer.step) + # self._summary_writer.add_scalar(name + "_std", param.std(), global_step=self.trainer.step) + self._summary_writer.add_scalar(name + "_grad_mean", param.grad.mean(), + global_step=self.trainer.step) + + def on_valid_end(self, eval_result, metric_key): + if "metric" in self.options: + for name, metric in eval_result.items(): + for metric_key, metric_val in metric.items(): + self._summary_writer.add_scalar("valid_{}_{}".format(name, metric_key), metric_val, + global_step=self.trainer.step) + + def on_train_end(self): + self._summary_writer.close() + del self._summary_writer + + def on_exception(self, exception): + if hasattr(self, "_summary_writer"): + self._summary_writer.close() + del self._summary_writer diff --git a/reproduction/__init__.py b/reproduction/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/reproduction/chinese_word_segment/cws_io/cws_reader.py b/reproduction/chinese_word_segment/cws_io/cws_reader.py deleted file mode 100644 index 34bcf7dd..00000000 --- a/reproduction/chinese_word_segment/cws_io/cws_reader.py +++ /dev/null @@ -1,197 +0,0 @@ - - -from fastNLP.core.dataset import DataSet -from fastNLP.core.instance import Instance -from fastNLP.io.dataset_loader import DataSetLoader - - -def cut_long_sentence(sent, max_sample_length=200): - """ - 将长于max_sample_length的sentence截成多段，只会在有空格的地方发生截断。所以截取的句子可能长于或者短于max_sample_length - - :param sent: str. - :param max_sample_length: int. - :return: list of str. - """ - sent_no_space = sent.replace(' ', '') - cutted_sentence = [] - if len(sent_no_space) > max_sample_length: - parts = sent.strip().split() - new_line = '' - length = 0 - for part in parts: - length += len(part) - new_line += part + ' ' - if length > max_sample_length: - new_line = new_line[:-1] - cutted_sentence.append(new_line) - length = 0 - new_line = '' - if new_line != '': - cutted_sentence.append(new_line[:-1]) - else: - cutted_sentence.append(sent) - return cutted_sentence - -class NaiveCWSReader(DataSetLoader): - """ - 这个reader假设了分词数据集为以下形式, 即已经用空格分割好内容了 - 这是 fastNLP , 一个 非常 good 的 包 . - 或者,即每个part后面还有一个pos tag - 也/D 在/P 團員/Na 之中/Ng ，/COMMACATEGORY - """ - def __init__(self, in_word_splitter=None): - super().__init__() - - self.in_word_splitter = in_word_splitter - - def load(self, filepath, in_word_splitter=None, cut_long_sent=False): - """ - 允许使用的情况有(默认以\t或空格作为seg) - 这是 fastNLP , 一个 非常 good 的 包 . - 和 - 也/D 在/P 團員/Na 之中/Ng ，/COMMACATEGORY - 如果splitter不为None则认为是第二种情况, 且我们会按splitter分割"也/D", 然后取第一部分. 例如"也/D".split('/')[0] - :param filepath: - :param in_word_splitter: - :return: - """ - if in_word_splitter == None: - in_word_splitter = self.in_word_splitter - dataset = DataSet() - with open(filepath, 'r') as f: - for line in f: - line = line.strip() - if len(line.replace(' ', ''))==0: # 不能接受空行 - continue - - if not in_word_splitter is None: - words = [] - for part in line.split(): - word = part.split(in_word_splitter)[0] - words.append(word) - line = ' '.join(words) - if cut_long_sent: - sents = cut_long_sentence(line) - else: - sents = [line] - for sent in sents: - instance = Instance(raw_sentence=sent) - dataset.append(instance) - - return dataset - - -class POSCWSReader(DataSetLoader): - """ - 支持读取以下的情况, 即每一行是一个词, 用空行作为两句话的界限. - 迈 N - 向 N - 充 N - ... - 泽 I-PER - 民 I-PER - - （ N - 一 N - 九 N - ... - - - :param filepath: - :return: - """ - def __init__(self, in_word_splitter=None): - super().__init__() - self.in_word_splitter = in_word_splitter - - def load(self, filepath, in_word_splitter=None, cut_long_sent=False): - if in_word_splitter is None: - in_word_splitter = self.in_word_splitter - dataset = DataSet() - with open(filepath, 'r') as f: - words = [] - for line in f: - line = line.strip() - if len(line) == 0: # new line - if len(words)==0: # 不能接受空行 - continue - line = ' '.join(words) - if cut_long_sent: - sents = cut_long_sentence(line) - else: - sents = [line] - for sent in sents: - instance = Instance(raw_sentence=sent) - dataset.append(instance) - words = [] - else: - line = line.split()[0] - if in_word_splitter is None: - words.append(line) - else: - words.append(line.split(in_word_splitter)[0]) - return dataset - - -class ConllCWSReader(object): - def __init__(self): - pass - - def load(self, path, cut_long_sent=False): - """ - 返回的DataSet只包含raw_sentence这个field，内容为str。 - 假定了输入为conll的格式，以空行隔开两个句子，每行共7列，即 - 1 编者按 编者按 NN O 11 nmod:topic - 2 ： ： PU O 11 punct - 3 7月 7月 NT DATE 4 compound:nn - 4 12日 12日 NT DATE 11 nmod:tmod - 5 ， ， PU O 11 punct - - 1 这 这 DT O 3 det - 2 款 款 M O 1 mark:clf - 3 飞行 飞行 NN O 8 nsubj - 4 从 从 P O 5 case - 5 外型 外型 NN O 8 nmod:prep - """ - datalist = [] - with open(path, 'r', encoding='utf-8') as f: - sample = [] - for line in f: - if line.startswith('\n'): - datalist.append(sample) - sample = [] - elif line.startswith('#'): - continue - else: - sample.append(line.split('\t')) - if len(sample) > 0: - datalist.append(sample) - - ds = DataSet() - for sample in datalist: - # print(sample) - res = self.get_char_lst(sample) - if res is None: - continue - line = ' '.join(res) - if cut_long_sent: - sents = cut_long_sentence(line) - else: - sents = [line] - for raw_sentence in sents: - ds.append(Instance(raw_sentence=raw_sentence)) - - return ds - - def get_char_lst(self, sample): - if len(sample)==0: - return None - text = [] - for w in sample: - t1, t2, t3, t4 = w[1], w[3], w[6], w[7] - if t3 == '_': - return None - text.append(t1) - return text - diff --git a/reproduction/chinese_word_segment/run.py b/reproduction/chinese_word_segment/run.py deleted file mode 100644 index e7804bae..00000000 --- a/reproduction/chinese_word_segment/run.py +++ /dev/null @@ -1,151 +0,0 @@ -import os -import sys - -sys.path.append(os.path.join(os.path.dirname(__file__), '../..')) - -from fastNLP.io.config_io import ConfigLoader, ConfigSection -from fastNLP.core.trainer import SeqLabelTrainer -from fastNLP.io.dataset_loader import BaseLoader, TokenizeDataSetLoader -from fastNLP.core.utils import load_pickle -from fastNLP.io.model_io import ModelLoader, ModelSaver -from fastNLP.core.tester import SeqLabelTester -from fastNLP.models.sequence_modeling import AdvSeqLabel -from fastNLP.core.predictor import SeqLabelInfer -from fastNLP.core.utils import save_pickle -from fastNLP.core.metrics import SeqLabelEvaluator - -# not in the file's dir -if len(os.path.dirname(__file__)) != 0: - os.chdir(os.path.dirname(__file__)) -datadir = "/home/zyfeng/data/" -cfgfile = './cws.cfg' - -cws_data_path = os.path.join(datadir, "pku_training.utf8") -pickle_path = "save" -data_infer_path = os.path.join(datadir, "infer.utf8") - - -def infer(): - # Config Loader - test_args = ConfigSection() - ConfigLoader().load_config(cfgfile, {"POS_test": test_args}) - - # fetch dictionary size and number of labels from pickle files - word2index = load_pickle(pickle_path, "word2id.pkl") - test_args["vocab_size"] = len(word2index) - index2label = load_pickle(pickle_path, "label2id.pkl") - test_args["num_classes"] = len(index2label) - - # Define the same model - model = AdvSeqLabel(test_args) - - try: - ModelLoader.load_pytorch(model, "./save/trained_model.pkl") - print('model loaded!') - except Exception as e: - print('cannot load model!') - raise - - # Data Loader - infer_data = SeqLabelDataSet(load_func=BaseLoader.load_lines) - infer_data.load(data_infer_path, vocabs={"word_vocab": word2index}, infer=True) - print('data loaded') - - # Inference interface - infer = SeqLabelInfer(pickle_path) - results = infer.predict(model, infer_data) - - print(results) - print("Inference finished!") - - -def train(): - # Config Loader - train_args = ConfigSection() - test_args = ConfigSection() - ConfigLoader().load_config(cfgfile, {"train": train_args, "test": test_args}) - - print("loading data set...") - data = SeqLabelDataSet(load_func=TokenizeDataSetLoader.load) - data.load(cws_data_path) - data_train, data_dev = data.split(ratio=0.3) - train_args["vocab_size"] = len(data.word_vocab) - train_args["num_classes"] = len(data.label_vocab) - print("vocab size={}, num_classes={}".format(len(data.word_vocab), len(data.label_vocab))) - - change_field_is_target(data_dev, "truth", True) - save_pickle(data_dev, "./save/", "data_dev.pkl") - save_pickle(data.word_vocab, "./save/", "word2id.pkl") - save_pickle(data.label_vocab, "./save/", "label2id.pkl") - - # Trainer - trainer = SeqLabelTrainer(epochs=train_args["epochs"], batch_size=train_args["batch_size"], - validate=train_args["validate"], - use_cuda=train_args["use_cuda"], pickle_path=train_args["pickle_path"], - save_best_dev=True, print_every_step=10, model_name="trained_model.pkl", - evaluator=SeqLabelEvaluator()) - - # Model - model = AdvSeqLabel(train_args) - try: - ModelLoader.load_pytorch(model, "./save/saved_model.pkl") - print('model parameter loaded!') - except Exception as e: - print("No saved model. Continue.") - pass - - # Start training - trainer.train(model, data_train, data_dev) - print("Training finished!") - - # Saver - saver = ModelSaver("./save/trained_model.pkl") - saver.save_pytorch(model) - print("Model saved!") - - -def predict(): - # Config Loader - test_args = ConfigSection() - ConfigLoader().load_config(cfgfile, {"POS_test": test_args}) - - # fetch dictionary size and number of labels from pickle files - word2index = load_pickle(pickle_path, "word2id.pkl") - test_args["vocab_size"] = len(word2index) - index2label = load_pickle(pickle_path, "label2id.pkl") - test_args["num_classes"] = len(index2label) - - # load dev data - dev_data = load_pickle(pickle_path, "data_dev.pkl") - - # Define the same model - model = AdvSeqLabel(test_args) - - # Dump trained parameters into the model - ModelLoader.load_pytorch(model, "./save/trained_model.pkl") - print("model loaded!") - - # Tester - test_args["evaluator"] = SeqLabelEvaluator() - tester = SeqLabelTester(**test_args.data) - - # Start testing - tester.test(model, dev_data) - - -if __name__ == "__main__": - - import argparse - - parser = argparse.ArgumentParser(description='Run a chinese word segmentation model') - parser.add_argument('--mode', help='set the model\'s model', choices=['train', 'test', 'infer']) - args = parser.parse_args() - if args.mode == 'train': - train() - elif args.mode == 'test': - predict() - elif args.mode == 'infer': - infer() - else: - print('no mode specified for model!') - parser.print_help() diff --git a/reproduction/pos_tag_model/pos_reader.py b/reproduction/pos_tag_model/pos_reader.py deleted file mode 100644 index c0a8c4cd..00000000 --- a/reproduction/pos_tag_model/pos_reader.py +++ /dev/null @@ -1,153 +0,0 @@ - -from fastNLP.core.dataset import DataSet -from fastNLP.core.instance import Instance - -def cut_long_sentence(sent, max_sample_length=200): - sent_no_space = sent.replace(' ', '') - cutted_sentence = [] - if len(sent_no_space) > max_sample_length: - parts = sent.strip().split() - new_line = '' - length = 0 - for part in parts: - length += len(part) - new_line += part + ' ' - if length > max_sample_length: - new_line = new_line[:-1] - cutted_sentence.append(new_line) - length = 0 - new_line = '' - if new_line != '': - cutted_sentence.append(new_line[:-1]) - else: - cutted_sentence.append(sent) - return cutted_sentence - - -class ConllPOSReader(object): - # 返回的Dataset包含words(list of list, 里层的list是character), tag两个field(list of str, str是标有BIO的tag)。 - def __init__(self): - pass - - def load(self, path): - datalist = [] - with open(path, 'r', encoding='utf-8') as f: - sample = [] - for line in f: - if line.startswith('\n'): - datalist.append(sample) - sample = [] - elif line.startswith('#'): - continue - else: - sample.append(line.split('\t')) - if len(sample) > 0: - datalist.append(sample) - - ds = DataSet() - for sample in datalist: - # print(sample) - res = self.get_one(sample) - if res is None: - continue - char_seq = [] - pos_seq = [] - for word, tag in zip(res[0], res[1]): - if len(word)==1: - char_seq.append(word) - pos_seq.append('S-{}'.format(tag)) - elif len(word)>1: - pos_seq.append('B-{}'.format(tag)) - for _ in range(len(word)-2): - pos_seq.append('M-{}'.format(tag)) - pos_seq.append('E-{}'.format(tag)) - char_seq.extend(list(word)) - else: - raise ValueError("Zero length of word detected.") - - ds.append(Instance(words=char_seq, - tag=pos_seq)) - - return ds - - - -class ZhConllPOSReader(object): - # 中文colln格式reader - def __init__(self): - pass - - def load(self, path): - """ - 返回的DataSet, 包含以下的field - words：list of str, - tag: list of str, 被加入了BMES tag, 比如原来的序列为['VP', 'NN', 'NN', ..]，会被认为是["S-VP", "B-NN", "M-NN",..] - 假定了输入为conll的格式，以空行隔开两个句子，每行共7列，即 - 1 编者按 编者按 NN O 11 nmod:topic - 2 ： ： PU O 11 punct - 3 7月 7月 NT DATE 4 compound:nn - 4 12日 12日 NT DATE 11 nmod:tmod - 5 ， ， PU O 11 punct - - 1 这 这 DT O 3 det - 2 款 款 M O 1 mark:clf - 3 飞行 飞行 NN O 8 nsubj - 4 从 从 P O 5 case - 5 外型 外型 NN O 8 nmod:prep - """ - datalist = [] - with open(path, 'r', encoding='utf-8') as f: - sample = [] - for line in f: - if line.startswith('\n'): - datalist.append(sample) - sample = [] - elif line.startswith('#'): - continue - else: - sample.append(line.split('\t')) - if len(sample) > 0: - datalist.append(sample) - - ds = DataSet() - for sample in datalist: - # print(sample) - res = self.get_one(sample) - if res is None: - continue - char_seq = [] - pos_seq = [] - for word, tag in zip(res[0], res[1]): - char_seq.extend(list(word)) - if len(word)==1: - pos_seq.append('S-{}'.format(tag)) - elif len(word)>1: - pos_seq.append('B-{}'.format(tag)) - for _ in range(len(word)-2): - pos_seq.append('M-{}'.format(tag)) - pos_seq.append('E-{}'.format(tag)) - else: - raise ValueError("Zero length of word detected.") - - ds.append(Instance(words=char_seq, - tag=pos_seq)) - - return ds - - def get_one(self, sample): - if len(sample)==0: - return None - text = [] - pos_tags = [] - for w in sample: - t1, t2, t3, t4 = w[1], w[3], w[6], w[7] - if t3 == '_': - return None - text.append(t1) - pos_tags.append(t2) - return text, pos_tags - -if __name__ == '__main__': - reader = ZhConllPOSReader() - d = reader.load('/home/hyan/train.conllx') - print(d) \ No newline at end of file diff --git a/reproduction/pos_tag_model/train_pos_tag.py b/reproduction/pos_tag_model/train_pos_tag.py deleted file mode 100644 index adc9359c..00000000 --- a/reproduction/pos_tag_model/train_pos_tag.py +++ /dev/null @@ -1,113 +0,0 @@ -import argparse -import os -import pickle -import sys - -import torch - -# in order to run fastNLP without installation -sys.path.append(os.path.join(os.path.dirname(__file__), '../..')) - - -from fastNLP.api.pipeline import Pipeline -from fastNLP.api.processor import SeqLenProcessor -from fastNLP.core.metrics import SpanFPreRecMetric -from fastNLP.core.trainer import Trainer -from fastNLP.io.config_io import ConfigLoader, ConfigSection -from fastNLP.models.sequence_modeling import AdvSeqLabel -from reproduction.chinese_word_segment.process.cws_processor import VocabIndexerProcessor -from reproduction.pos_tag_model.pos_reader import ZhConllPOSReader -from fastNLP.api.processor import ModelProcessor, Index2WordProcessor - -cfgfile = './pos_tag.cfg' -pickle_path = "save" - - -def load_tencent_embed(embed_path, word2id): - hit = 0 - with open(embed_path, "rb") as f: - embed_dict = pickle.load(f) - embedding_tensor = torch.randn(len(word2id), 200) - for key in word2id: - if key in embed_dict: - embedding_tensor[word2id[key]] = torch.Tensor(embed_dict[key]) - hit += 1 - print("vocab_size={} hit={} hit/vocab_size={}".format(len(word2id), hit, hit / len(word2id))) - return embedding_tensor - - -def train(checkpoint=None): - # load config - train_param = ConfigSection() - model_param = ConfigSection() - ConfigLoader().load_config(cfgfile, {"train": train_param, "model": model_param}) - print("config loaded") - - # Data Loader - dataset = ZhConllPOSReader().load("/home/hyan/train.conllx") - print(dataset) - print("dataset transformed") - - dataset.rename_field("tag", "truth") - - vocab_proc = VocabIndexerProcessor("words", new_added_filed_name="word_seq") - tag_proc = VocabIndexerProcessor("truth") - seq_len_proc = SeqLenProcessor(field_name="word_seq", new_added_field_name="word_seq_origin_len", is_input=True) - - vocab_proc(dataset) - tag_proc(dataset) - seq_len_proc(dataset) - - dataset.set_input("word_seq", "word_seq_origin_len", "truth") - dataset.set_target("truth", "word_seq_origin_len") - - print("processors defined") - - # dataset.set_is_target(tag_ids=True) - model_param["vocab_size"] = vocab_proc.get_vocab_size() - model_param["num_classes"] = tag_proc.get_vocab_size() - print("vocab_size={} num_classes={}".format(model_param["vocab_size"], model_param["num_classes"])) - - # define a model - if checkpoint is None: - # pre_trained = load_tencent_embed("/home/zyfeng/data/char_tencent_embedding.pkl", vocab_proc.vocab.word2idx) - pre_trained = None - model = AdvSeqLabel(model_param, id2words=tag_proc.vocab.idx2word, emb=pre_trained) - print(model) - else: - model = torch.load(checkpoint) - - # call trainer to train - trainer = Trainer(dataset, model, loss=None, metrics=SpanFPreRecMetric(tag_proc.vocab, pred="predict", - target="truth", - seq_lens="word_seq_origin_len"), - dev_data=dataset, metric_key="f", - use_tqdm=True, use_cuda=True, print_every=5, n_epochs=6, save_path="./save") - trainer.train(load_best_model=True) - - # save model & pipeline - model_proc = ModelProcessor(model, seq_len_field_name="word_seq_origin_len") - id2tag = Index2WordProcessor(tag_proc.vocab, "predict", "tag") - - pp = Pipeline([vocab_proc, seq_len_proc, model_proc, id2tag]) - save_dict = {"pipeline": pp, "model": model, "tag_vocab": tag_proc.vocab} - torch.save(save_dict, "model_pp.pkl") - print("pipeline saved") - - torch.save(model, "./save/best_model.pkl") - - -if __name__ == "__main__": - parser = argparse.ArgumentParser() - parser.add_argument("-c", "--restart", action="store_true", help="whether to continue training") - parser.add_argument("-cp", "--checkpoint", type=str, help="checkpoint of the trained model") - args = parser.parse_args() - - if args.restart is True: - # 继续训练 python train_pos_tag.py -c -cp ./save/best_model.pkl - if args.checkpoint is None: - raise RuntimeError("Please provide the checkpoint. -cp ") - train(args.checkpoint) - else: - # 一次训练 python train_pos_tag.py - train() diff --git a/requirements.txt b/requirements.txt index 45c84bc2..dfd2b16e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -numpy>=1.14.2 +numpy torch>=0.4.0 -tensorboardX -tqdm>=4.28.1 \ No newline at end of file +tqdm +nltk \ No newline at end of file diff --git a/setup.py b/setup.py index a8b4834e..49646761 100644 --- a/setup.py +++ b/setup.py @@ -13,12 +13,13 @@ with open('requirements.txt', encoding='utf-8') as f: setup( name='FastNLP', - version='0.1.1', + version='0.4.0', description='fastNLP: Deep Learning Toolkit for NLP, developed by Fudan FastNLP Team', long_description=readme, - license=license, + long_description_content_type='text/markdown', + license='Apache License', author='FudanNLP', - python_requires='>=3.5', + python_requires='>=3.6', packages=find_packages(), install_requires=reqs.strip().split('\n'), ) diff --git a/test/api/test_pipeline.py b/test/api/test_pipeline.py deleted file mode 100644 index c7094790..00000000 --- a/test/api/test_pipeline.py +++ /dev/null @@ -1,6 +0,0 @@ -import unittest - - -class TestPipeline(unittest.TestCase): - def test_case(self): - pass diff --git a/test/api/test_processor.py b/test/api/test_processor.py deleted file mode 100644 index f515e507..00000000 --- a/test/api/test_processor.py +++ /dev/null @@ -1,55 +0,0 @@ -import random -import unittest - -from fastNLP import Vocabulary -from fastNLP.api.processor import FullSpaceToHalfSpaceProcessor, PreAppendProcessor, SliceProcessor, Num2TagProcessor, \ - IndexerProcessor, VocabProcessor, SeqLenProcessor -from fastNLP.core.dataset import DataSet - - -class TestProcessor(unittest.TestCase): - def test_FullSpaceToHalfSpaceProcessor(self): - ds = DataSet({"word": ["０0, u１, u), (u２, u2"]}) - proc = FullSpaceToHalfSpaceProcessor("word") - ds = proc(ds) - self.assertEqual(ds.field_arrays["word"].content, ["00, u1, u), (u2, u2"]) - - def test_PreAppendProcessor(self): - ds = DataSet({"word": [["1234", "3456"], ["8789", "3464"]]}) - proc = PreAppendProcessor(data="abc", field_name="word") - ds = proc(ds) - self.assertEqual(ds.field_arrays["word"].content, [["abc", "1234", "3456"], ["abc", "8789", "3464"]]) - - def test_SliceProcessor(self): - ds = DataSet({"xx": [[random.randint(0, 10) for _ in range(30)]] * 40}) - proc = SliceProcessor(10, 20, 2, "xx", new_added_field_name="yy") - ds = proc(ds) - self.assertEqual(len(ds.field_arrays["yy"].content[0]), 5) - - def test_Num2TagProcessor(self): - ds = DataSet({"num": [["99.9982", "2134.0"], ["0.002", "234"]]}) - proc = Num2TagProcessor("", "num") - ds = proc(ds) - for data in ds.field_arrays["num"].content: - for d in data: - self.assertEqual(d, "") - - def test_VocabProcessor_and_IndexerProcessor(self): - ds = DataSet({"xx": [[str(random.randint(0, 10)) for _ in range(30)]] * 40}) - vocab_proc = VocabProcessor("xx") - vocab_proc(ds) - vocab = vocab_proc.vocab - self.assertTrue(isinstance(vocab, Vocabulary)) - self.assertTrue(len(vocab) > 5) - - proc = IndexerProcessor(vocab, "xx", "yy") - ds = proc(ds) - for data in ds.field_arrays["yy"].content[0]: - self.assertTrue(isinstance(data, int)) - - def test_SeqLenProcessor(self): - ds = DataSet({"xx": [[str(random.randint(0, 10)) for _ in range(30)]] * 10}) - proc = SeqLenProcessor("xx", "len") - ds = proc(ds) - for data in ds.field_arrays["len"].content: - self.assertEqual(data, 30) diff --git a/test/core/test_batch.py b/test/core/test_batch.py index 08d803f1..d1f93b9c 100644 --- a/test/core/test_batch.py +++ b/test/core/test_batch.py @@ -1,11 +1,55 @@ import unittest import numpy as np +import torch -from fastNLP.core.batch import Batch -from fastNLP.core.dataset import DataSet -from fastNLP.core.dataset import construct_dataset -from fastNLP.core.sampler import SequentialSampler +from fastNLP import Batch +from fastNLP import DataSet +from fastNLP import Instance +from fastNLP import SequentialSampler + + +def generate_fake_dataset(num_samples=1000): + """ + 产生的DataSet包含以下的field {'1':[], '2':[], '3': [], '4':[]} + :param num_samples: sample的数量 + :return: + """ + + max_len = 50 + min_len = 10 + num_features = 4 + + data_dict = {} + for i in range(num_features): + data = [] + lengths = np.random.randint(min_len, max_len, size=(num_samples)) + for length in lengths: + data.append(np.random.randint(100, size=length)) + data_dict[str(i)] = data + + dataset = DataSet(data_dict) + + for i in range(num_features): + if np.random.randint(2) == 0: + dataset.set_input(str(i)) + else: + dataset.set_target(str(i)) + return dataset + + +def construct_dataset(sentences): + """Construct a data set from a list of sentences. + + :param sentences: list of list of str + :return dataset: a DataSet object + """ + dataset = DataSet() + for sentence in sentences: + instance = Instance() + instance['raw_sentence'] = sentence + dataset.append(instance) + return dataset class TestCase1(unittest.TestCase): @@ -14,12 +58,12 @@ class TestCase1(unittest.TestCase): [["FastNLP", "is", "the", "most", "beautiful", "tool", "in", "the", "world"] for _ in range(40)]) dataset.set_target() batch = Batch(dataset, batch_size=4, sampler=SequentialSampler(), as_numpy=True) - + cnt = 0 for _, _ in batch: cnt += 1 self.assertEqual(cnt, 10) - + def test_dataset_batching(self): ds = DataSet({"x": [[1, 2, 3, 4]] * 40, "y": [[5, 6]] * 40}) ds.set_input("x") @@ -31,3 +75,115 @@ class TestCase1(unittest.TestCase): self.assertEqual(len(y["y"]), 4) self.assertListEqual(list(x["x"][-1]), [1, 2, 3, 4]) self.assertListEqual(list(y["y"][-1]), [5, 6]) + + def test_list_padding(self): + ds = DataSet({"x": [[1], [1, 2], [1, 2, 3], [1, 2, 3, 4]] * 10, + "y": [[4, 3, 2, 1], [3, 2, 1], [2, 1], [1]] * 10}) + ds.set_input("x") + ds.set_target("y") + iter = Batch(ds, batch_size=4, sampler=SequentialSampler(), as_numpy=True) + for x, y in iter: + self.assertEqual(x["x"].shape, (4, 4)) + self.assertEqual(y["y"].shape, (4, 4)) + + def test_numpy_padding(self): + ds = DataSet({"x": np.array([[1], [1, 2], [1, 2, 3], [1, 2, 3, 4]] * 10), + "y": np.array([[4, 3, 2, 1], [3, 2, 1], [2, 1], [1]] * 10)}) + ds.set_input("x") + ds.set_target("y") + iter = Batch(ds, batch_size=4, sampler=SequentialSampler(), as_numpy=True) + for x, y in iter: + self.assertEqual(x["x"].shape, (4, 4)) + self.assertEqual(y["y"].shape, (4, 4)) + + def test_list_to_tensor(self): + ds = DataSet({"x": [[1], [1, 2], [1, 2, 3], [1, 2, 3, 4]] * 10, + "y": [[4, 3, 2, 1], [3, 2, 1], [2, 1], [1]] * 10}) + ds.set_input("x") + ds.set_target("y") + iter = Batch(ds, batch_size=4, sampler=SequentialSampler(), as_numpy=False) + for x, y in iter: + self.assertTrue(isinstance(x["x"], torch.Tensor)) + self.assertEqual(tuple(x["x"].shape), (4, 4)) + self.assertTrue(isinstance(y["y"], torch.Tensor)) + self.assertEqual(tuple(y["y"].shape), (4, 4)) + + def test_numpy_to_tensor(self): + ds = DataSet({"x": np.array([[1], [1, 2], [1, 2, 3], [1, 2, 3, 4]] * 10), + "y": np.array([[4, 3, 2, 1], [3, 2, 1], [2, 1], [1]] * 10)}) + ds.set_input("x") + ds.set_target("y") + iter = Batch(ds, batch_size=4, sampler=SequentialSampler(), as_numpy=False) + for x, y in iter: + self.assertTrue(isinstance(x["x"], torch.Tensor)) + self.assertEqual(tuple(x["x"].shape), (4, 4)) + self.assertTrue(isinstance(y["y"], torch.Tensor)) + self.assertEqual(tuple(y["y"].shape), (4, 4)) + + def test_list_of_list_to_tensor(self): + ds = DataSet([Instance(x=[1, 2], y=[3, 4]) for _ in range(2)] + + [Instance(x=[1, 2, 3, 4], y=[3, 4, 5, 6]) for _ in range(2)]) + ds.set_input("x") + ds.set_target("y") + iter = Batch(ds, batch_size=4, sampler=SequentialSampler(), as_numpy=False) + for x, y in iter: + self.assertTrue(isinstance(x["x"], torch.Tensor)) + self.assertEqual(tuple(x["x"].shape), (4, 4)) + self.assertTrue(isinstance(y["y"], torch.Tensor)) + self.assertEqual(tuple(y["y"].shape), (4, 4)) + + def test_list_of_numpy_to_tensor(self): + ds = DataSet([Instance(x=np.array([1, 2]), y=np.array([3, 4])) for _ in range(2)] + + [Instance(x=np.array([1, 2, 3, 4]), y=np.array([3, 4, 5, 6])) for _ in range(2)]) + ds.set_input("x") + ds.set_target("y") + iter = Batch(ds, batch_size=4, sampler=SequentialSampler(), as_numpy=False) + for x, y in iter: + print(x, y) + + def test_sequential_batch(self): + batch_size = 32 + num_samples = 1000 + dataset = generate_fake_dataset(num_samples) + + batch = Batch(dataset, batch_size=batch_size, sampler=SequentialSampler()) + for batch_x, batch_y in batch: + pass + + """ + def test_multi_workers_batch(self): + batch_size = 32 + pause_seconds = 0.01 + num_samples = 1000 + dataset = generate_fake_dataset(num_samples) + + num_workers = 1 + batch = Batch(dataset, batch_size=batch_size, sampler=SequentialSampler(), num_workers=num_workers) + for batch_x, batch_y in batch: + time.sleep(pause_seconds) + + num_workers = 2 + batch = Batch(dataset, batch_size=batch_size, sampler=SequentialSampler(), num_workers=num_workers) + end1 = time.time() + for batch_x, batch_y in batch: + time.sleep(pause_seconds) + """ + """ + def test_pin_memory(self): + batch_size = 32 + pause_seconds = 0.01 + num_samples = 1000 + dataset = generate_fake_dataset(num_samples) + + batch = Batch(dataset, batch_size=batch_size, sampler=SequentialSampler(), pin_memory=True) + # 这里发生OOM + # for batch_x, batch_y in batch: + # time.sleep(pause_seconds) + + num_workers = 2 + batch = Batch(dataset, batch_size=batch_size, sampler=SequentialSampler(), num_workers=num_workers, + pin_memory=True) + # 这里发生OOM + # for batch_x, batch_y in batch: + # time.sleep(pause_seconds) + """ diff --git a/test/core/test_callbacks.py b/test/core/test_callbacks.py index 20822cde..db640eb1 100644 --- a/test/core/test_callbacks.py +++ b/test/core/test_callbacks.py @@ -1,44 +1,155 @@ import unittest import numpy as np +import torch -from fastNLP.core.callback import EchoCallback -from fastNLP.core.dataset import DataSet -from fastNLP.core.instance import Instance -from fastNLP.core.losses import BCELoss -from fastNLP.core.optimizer import SGD -from fastNLP.core.trainer import Trainer +from fastNLP.core.callback import EarlyStopCallback, GradientClipCallback, LRScheduler, ControlC, \ + LRFinder, TensorboardCallback +from fastNLP import DataSet +from fastNLP import Instance +from fastNLP import BCELoss +from fastNLP import AccuracyMetric +from fastNLP import SGD +from fastNLP import Trainer from fastNLP.models.base_model import NaiveClassifier -class TestCallback(unittest.TestCase): - def test_case(self): - def prepare_fake_dataset(): - mean = np.array([-3, -3]) - cov = np.array([[1, 0], [0, 1]]) - class_A = np.random.multivariate_normal(mean, cov, size=(1000,)) - - mean = np.array([3, 3]) - cov = np.array([[1, 0], [0, 1]]) - class_B = np.random.multivariate_normal(mean, cov, size=(1000,)) - - data_set = DataSet([Instance(x=[float(item[0]), float(item[1])], y=[0.0]) for item in class_A] + - [Instance(x=[float(item[0]), float(item[1])], y=[1.0]) for item in class_B]) - return data_set - - data_set = prepare_fake_dataset() - data_set.set_input("x") - data_set.set_target("y") +def prepare_env(): + def prepare_fake_dataset(): + mean = np.array([-3, -3]) + cov = np.array([[1, 0], [0, 1]]) + class_A = np.random.multivariate_normal(mean, cov, size=(1000,)) + + mean = np.array([3, 3]) + cov = np.array([[1, 0], [0, 1]]) + class_B = np.random.multivariate_normal(mean, cov, size=(1000,)) + + data_set = DataSet([Instance(x=[float(item[0]), float(item[1])], y=[0.0]) for item in class_A] + + [Instance(x=[float(item[0]), float(item[1])], y=[1.0]) for item in class_B]) + return data_set + + data_set = prepare_fake_dataset() + data_set.set_input("x") + data_set.set_target("y") + model = NaiveClassifier(2, 1) + return data_set, model - model = NaiveClassifier(2, 1) +class TestCallback(unittest.TestCase): + + def test_gradient_clip(self): + data_set, model = prepare_env() + trainer = Trainer(data_set, model, + loss=BCELoss(pred="predict", target="y"), + n_epochs=20, + batch_size=32, + print_every=50, + optimizer=SGD(lr=0.1), + check_code_level=2, + use_tqdm=False, + dev_data=data_set, + metrics=AccuracyMetric(pred="predict", target="y"), + callbacks=[GradientClipCallback(model.parameters(), clip_value=2)]) + trainer.train() + + def test_early_stop(self): + data_set, model = prepare_env() + trainer = Trainer(data_set, model, + loss=BCELoss(pred="predict", target="y"), + n_epochs=20, + batch_size=32, + print_every=50, + optimizer=SGD(lr=0.01), + check_code_level=2, + use_tqdm=False, + dev_data=data_set, + metrics=AccuracyMetric(pred="predict", target="y"), + callbacks=[EarlyStopCallback(5)]) + trainer.train() + + def test_lr_scheduler(self): + data_set, model = prepare_env() + optimizer = torch.optim.SGD(model.parameters(), lr=0.01) + trainer = Trainer(data_set, model, + loss=BCELoss(pred="predict", target="y"), + n_epochs=5, + batch_size=32, + print_every=50, + optimizer=optimizer, + check_code_level=2, + use_tqdm=False, + dev_data=data_set, + metrics=AccuracyMetric(pred="predict", target="y"), + callbacks=[LRScheduler(torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1))]) + trainer.train() + + def test_KeyBoardInterrupt(self): + data_set, model = prepare_env() + trainer = Trainer(data_set, model, + loss=BCELoss(pred="predict", target="y"), + n_epochs=5, + batch_size=32, + print_every=50, + optimizer=SGD(lr=0.1), + check_code_level=2, + use_tqdm=False, + callbacks=[ControlC(False)]) + trainer.train() + + def test_LRFinder(self): + data_set, model = prepare_env() + trainer = Trainer(data_set, model, + loss=BCELoss(pred="predict", target="y"), + n_epochs=5, + batch_size=32, + print_every=50, + optimizer=SGD(lr=0.1), + check_code_level=2, + use_tqdm=False, + callbacks=[LRFinder(len(data_set) // 32)]) + trainer.train() + + def test_TensorboardCallback(self): + data_set, model = prepare_env() + trainer = Trainer(data_set, model, + loss=BCELoss(pred="predict", target="y"), + n_epochs=5, + batch_size=32, + print_every=50, + optimizer=SGD(lr=0.1), + check_code_level=2, + use_tqdm=False, + dev_data=data_set, + metrics=AccuracyMetric(pred="predict", target="y"), + callbacks=[TensorboardCallback("loss", "metric")]) + trainer.train() + + def test_readonly_property(self): + from fastNLP.core.callback import Callback + passed_epochs = [] + total_epochs = 5 + + class MyCallback(Callback): + def __init__(self): + super(MyCallback, self).__init__() + + def on_epoch_begin(self): + passed_epochs.append(self.epoch) + print(self.n_epochs, self.n_steps, self.batch_size) + print(self.model) + print(self.optimizer) + + data_set, model = prepare_env() trainer = Trainer(data_set, model, loss=BCELoss(pred="predict", target="y"), - n_epochs=1, + n_epochs=total_epochs, batch_size=32, print_every=50, optimizer=SGD(lr=0.1), check_code_level=2, use_tqdm=False, - callbacks=[EchoCallback()]) + dev_data=data_set, + metrics=AccuracyMetric(pred="predict", target="y"), + callbacks=[MyCallback()]) trainer.train() + assert passed_epochs == list(range(1, total_epochs + 1)) diff --git a/test/core/test_dataset.py b/test/core/test_dataset.py index 261d42b3..0228f207 100644 --- a/test/core/test_dataset.py +++ b/test/core/test_dataset.py @@ -1,20 +1,35 @@ import os import unittest -from fastNLP.core.dataset import DataSet -from fastNLP.core.fieldarray import FieldArray -from fastNLP.core.instance import Instance - - -class TestDataSet(unittest.TestCase): - +from fastNLP import DataSet +from fastNLP import FieldArray +from fastNLP import Instance +from fastNLP.io import CSVLoader + + +class TestDataSetInit(unittest.TestCase): + """初始化DataSet的办法有以下几种： + 1) 用dict: + 1.1) 二维list DataSet({"x": [[1, 2], [3, 4]]}) + 1.2) 二维array DataSet({"x": np.array([[1, 2], [3, 4]])}) + 1.3) 三维list DataSet({"x": [[[1, 2], [3, 4]], [[1, 2], [3, 4]]]}) + 2) 用list of Instance: + 2.1) 一维list DataSet([Instance(x=[1, 2, 3, 4])]) + 2.2) 一维array DataSet([Instance(x=np.array([1, 2, 3, 4]))]) + 2.3) 二维list DataSet([Instance(x=[[1, 2], [3, 4]])]) + 2.4) 二维array DataSet([Instance(x=np.array([[1, 2], [3, 4]]))]) + + 只接受纯list或者最外层ndarray + """ def test_init_v1(self): + # 一维list ds = DataSet([Instance(x=[1, 2, 3, 4], y=[5, 6])] * 40) self.assertTrue("x" in ds.field_arrays and "y" in ds.field_arrays) self.assertEqual(ds.field_arrays["x"].content, [[1, 2, 3, 4], ] * 40) self.assertEqual(ds.field_arrays["y"].content, [[5, 6], ] * 40) def test_init_v2(self): + # 用dict ds = DataSet({"x": [[1, 2, 3, 4]] * 40, "y": [[5, 6]] * 40}) self.assertTrue("x" in ds.field_arrays and "y" in ds.field_arrays) self.assertEqual(ds.field_arrays["x"].content, [[1, 2, 3, 4], ] * 40) @@ -28,6 +43,8 @@ class TestDataSet(unittest.TestCase): with self.assertRaises(ValueError): _ = DataSet(0.00001) + +class TestDataSetMethods(unittest.TestCase): def test_append(self): dd = DataSet() for _ in range(3): @@ -36,7 +53,7 @@ class TestDataSet(unittest.TestCase): self.assertEqual(dd.field_arrays["x"].content, [[1, 2, 3, 4]] * 3) self.assertEqual(dd.field_arrays["y"].content, [[5, 6]] * 3) - def test_add_append(self): + def test_add_field(self): dd = DataSet() dd.add_field("x", [[1, 2, 3]] * 10) dd.add_field("y", [[1, 2, 3, 4]] * 10) @@ -49,6 +66,11 @@ class TestDataSet(unittest.TestCase): with self.assertRaises(RuntimeError): dd.add_field("??", [[1, 2]] * 40) + def test_add_field_ignore_type(self): + dd = DataSet() + dd.add_field("x", [(1, "1"), (2, "2"), (3, "3"), (4, "4")], ignore_type=True, is_target=True) + dd.add_field("y", [{1, "1"}, {2, "2"}, {3, "3"}, {4, "4"}], ignore_type=True, is_target=True) + def test_delete_field(self): dd = DataSet() dd.add_field("x", [[1, 2, 3]] * 10) @@ -99,9 +121,12 @@ class TestDataSet(unittest.TestCase): self.assertTrue(isinstance(res, list) and len(res) > 0) self.assertTrue(res[0], 4) + ds.apply(lambda ins: (len(ins["x"]), "hahaha"), new_field_name="k", ignore_type=True) + # expect no exception raised + def test_drop(self): ds = DataSet({"x": [[1, 2, 3, 4]] * 40, "y": [[5, 6], [7, 8, 9, 0]] * 20}) - ds.drop(lambda ins: len(ins["y"]) < 3) + ds.drop(lambda ins: len(ins["y"]) < 3, inplace=True) self.assertEqual(len(ds), 20) def test_contains(self): @@ -139,17 +164,20 @@ class TestDataSet(unittest.TestCase): ds = DataSet({"x": [[1, 2, 3, 4]] * 10, "y": [[5, 6]] * 10}) self.assertEqual(ds.get_target_name(), [_ for _ in ds.field_arrays if ds.field_arrays[_].is_target]) + def test_split(self): + ds = DataSet({"x": [[1, 2, 3, 4]] * 10, "y": [[5, 6]] * 10}) + d1, d2 = ds.split(0.1) + def test_apply2(self): def split_sent(ins): return ins['raw_sentence'].split() - - dataset = DataSet.read_csv('test/data_for_tests/tutorial_sample_dataset.csv', headers=('raw_sentence', 'label'), - sep='\t') - dataset.drop(lambda x: len(x['raw_sentence'].split()) == 0) + csv_loader = CSVLoader(headers=['raw_sentence', 'label'],sep='\t') + dataset = csv_loader.load('test/data_for_tests/tutorial_sample_dataset.csv') + dataset.drop(lambda x: len(x['raw_sentence'].split()) == 0, inplace=True) dataset.apply(split_sent, new_field_name='words', is_input=True) # print(dataset) - def test_add_field(self): + def test_add_field_v2(self): ds = DataSet({"x": [3, 4]}) ds.add_field('y', [['hello', 'world'], ['this', 'is', 'a', 'test']], is_input=True, is_target=True) # ds.apply(lambda x:[x['x']]*3, is_input=True, is_target=True, new_field_name='y') @@ -178,19 +206,11 @@ class TestDataSet(unittest.TestCase): self.assertTrue(isinstance(ans, FieldArray)) self.assertEqual(ans.content, [[5, 6]] * 10) - def test_reader(self): - # 跑通即可 - ds = DataSet().read_naive("test/data_for_tests/tutorial_sample_dataset.csv") - self.assertTrue(isinstance(ds, DataSet)) - self.assertTrue(len(ds) > 0) - - ds = DataSet().read_rawdata("test/data_for_tests/people_daily_raw.txt") - self.assertTrue(isinstance(ds, DataSet)) - self.assertTrue(len(ds) > 0) - - ds = DataSet().read_pos("test/data_for_tests/people.txt") - self.assertTrue(isinstance(ds, DataSet)) - self.assertTrue(len(ds) > 0) + def test_add_null(self): + # TODO test failed because 'fastNLP\core\field.py:143: RuntimeError' + ds = DataSet() + with self.assertRaises(RuntimeError) as RE: + ds.add_field('test', []) class TestDataSetIter(unittest.TestCase): diff --git a/test/core/test_field.py b/test/core/test_field.py new file mode 100644 index 00000000..1f6580c1 --- /dev/null +++ b/test/core/test_field.py @@ -0,0 +1,235 @@ +import unittest + +import numpy as np + +from fastNLP import FieldArray + + +class TestFieldArrayInit(unittest.TestCase): + """ + 1） 如果DataSet使用dict初始化，那么在add_field中会构造FieldArray： + 1.1) 二维list DataSet({"x": [[1, 2], [3, 4]]}) + 1.2) 二维array DataSet({"x": np.array([[1, 2], [3, 4]])}) + 1.3) 三维list DataSet({"x": [[[1, 2], [3, 4]], [[1, 2], [3, 4]]]}) + 2） 如果DataSet使用list of Instance 初始化,那么在append中会先对第一个样本初始化FieldArray； + 然后后面的样本使用FieldArray.append进行添加。 + 2.1) 一维list DataSet([Instance(x=[1, 2, 3, 4])]) + 2.2) 一维array DataSet([Instance(x=np.array([1, 2, 3, 4]))]) + 2.3) 二维list DataSet([Instance(x=[[1, 2], [3, 4]])]) + 2.4) 二维array DataSet([Instance(x=np.array([[1, 2], [3, 4]]))]) + """ + + def test_init_v1(self): + # 二维list + fa = FieldArray("x", [[1, 2], [3, 4]] * 5, is_input=True) + + def test_init_v2(self): + # 二维array + fa = FieldArray("x", np.array([[1, 2], [3, 4]] * 5), is_input=True) + + def test_init_v3(self): + # 三维list + fa = FieldArray("x", [[[1, 2], [3, 4]], [[1, 2], [3, 4]]], is_input=True) + + def test_init_v7(self): + # list of array + fa = FieldArray("x", [np.array([[1, 2], [3, 4]]), np.array([[1, 2], [3, 4]])], is_input=True) + self.assertEqual(fa.pytype, int) + self.assertEqual(fa.dtype, np.int) + + def test_init_v4(self): + # 一维list + val = [1, 2, 3, 4] + fa = FieldArray("x", [val], is_input=True) + fa.append(val) + + def test_init_v5(self): + # 一维array + val = np.array([1, 2, 3, 4]) + fa = FieldArray("x", [val], is_input=True) + fa.append(val) + + def test_init_v6(self): + # 二维array + val = [[1, 2], [3, 4]] + fa = FieldArray("x", [val], is_input=True) + fa.append(val) + + def test_init_v7(self): + # 二维list + val = np.array([[1, 2], [3, 4]]) + fa = FieldArray("x", [val], is_input=True) + fa.append(val) + + +class TestFieldArray(unittest.TestCase): + def test_main(self): + fa = FieldArray("x", [1, 2, 3, 4, 5], is_input=True) + self.assertEqual(len(fa), 5) + fa.append(6) + self.assertEqual(len(fa), 6) + + self.assertEqual(fa[-1], 6) + self.assertEqual(fa[0], 1) + fa[-1] = 60 + self.assertEqual(fa[-1], 60) + + self.assertEqual(fa.get(0), 1) + self.assertTrue(isinstance(fa.get([0, 1, 2]), np.ndarray)) + self.assertListEqual(list(fa.get([0, 1, 2])), [1, 2, 3]) + + def test_type_conversion(self): + fa = FieldArray("x", [1.2, 2.2, 3, 4, 5], is_input=True) + self.assertEqual(fa.pytype, float) + self.assertEqual(fa.dtype, np.float64) + + fa = FieldArray("x", [1, 2, 3, 4, 5], is_input=True) + fa.append(1.3333) + self.assertEqual(fa.pytype, float) + self.assertEqual(fa.dtype, np.float64) + + fa = FieldArray("y", [1.1, 2.2, 3.3, 4.4, 5.5], is_input=True) + fa.append(10) + self.assertEqual(fa.pytype, float) + self.assertEqual(fa.dtype, np.float64) + + fa = FieldArray("y", ["a", "b", "c", "d"], is_input=True) + fa.append("e") + self.assertEqual(fa.dtype, np.str) + self.assertEqual(fa.pytype, str) + + def test_support_np_array(self): + fa = FieldArray("y", np.array([[1.1, 2.2, 3.3, 4.4, 5.5]]), is_input=True) + self.assertEqual(fa.dtype, np.float64) + self.assertEqual(fa.pytype, float) + + fa.append(np.array([1.1, 2.2, 3.3, 4.4, 5.5])) + self.assertEqual(fa.dtype, np.float64) + self.assertEqual(fa.pytype, float) + + fa = FieldArray("my_field", np.random.rand(3, 5), is_input=True) + # in this case, pytype is actually a float. We do not care about it. + self.assertEqual(fa.dtype, np.float64) + + def test_nested_list(self): + fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1.1, 2.2, 3.3, 4.4, 5.5]], is_input=True) + self.assertEqual(fa.pytype, float) + self.assertEqual(fa.dtype, np.float64) + + def test_getitem_v1(self): + fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) + self.assertEqual(fa[0], [1.1, 2.2, 3.3, 4.4, 5.5]) + ans = fa[[0, 1]] + self.assertTrue(isinstance(ans, np.ndarray)) + self.assertTrue(isinstance(ans[0], np.ndarray)) + self.assertEqual(ans[0].tolist(), [1.1, 2.2, 3.3, 4.4, 5.5]) + self.assertEqual(ans[1].tolist(), [1, 2, 3, 4, 5]) + self.assertEqual(ans.dtype, np.float64) + + def test_getitem_v2(self): + x = np.random.rand(10, 5) + fa = FieldArray("my_field", x, is_input=True) + indices = [0, 1, 3, 4, 6] + for a, b in zip(fa[indices], x[indices]): + self.assertListEqual(a.tolist(), b.tolist()) + + def test_append(self): + with self.assertRaises(Exception): + fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) + fa.append(0) + + with self.assertRaises(Exception): + fa = FieldArray("y", [1.1, 2.2, 3.3, 4.4, 5.5], is_input=True) + fa.append([1, 2, 3, 4, 5]) + + with self.assertRaises(Exception): + fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) + fa.append([]) + + with self.assertRaises(Exception): + fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) + fa.append(["str", 0, 0, 0, 1.89]) + + fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) + fa.append([1.2, 2.3, 3.4, 4.5, 5.6]) + self.assertEqual(len(fa), 3) + self.assertEqual(fa[2], [1.2, 2.3, 3.4, 4.5, 5.6]) + + def test_ignore_type(self): + # 测试新添加的参数ignore_type，用来跳过类型检查 + fa = FieldArray("y", [[1.1, 2.2, "jin", {}, "hahah"], [int, 2, "$", 4, 5]], is_input=True, ignore_type=True) + fa.append([1.2, 2.3, str, 4.5, print]) + + fa = FieldArray("y", [(1, "1"), (2, "2"), (3, "3"), (4, "4")], is_target=True, ignore_type=True) + + +class TestPadder(unittest.TestCase): + + def test01(self): + """ + 测试AutoPadder能否正常工作 + :return: + """ + from fastNLP import AutoPadder + padder = AutoPadder() + content = ['This is a str', 'this is another str'] + self.assertListEqual(content, padder(content, None, np.str).tolist()) + + content = [1, 2] + self.assertListEqual(content, padder(content, None, np.int64).tolist()) + + content = [[1,2], [3], [4]] + self.assertListEqual([[1,2], [3, 0], [4, 0]], + padder(content, None, np.int64).tolist()) + + content = [ + [[1, 2, 3], [4, 5], [7,8,9,10]], + [[1]] + ] + self.assertListEqual(content, + padder(content, None, np.int64).tolist()) + + def test02(self): + """ + 测试EngChar2DPadder能不能正确使用 + :return: + """ + from fastNLP import EngChar2DPadder + padder = EngChar2DPadder(pad_length=0) + + contents = [1, 2] + # 不能是1维 + with self.assertRaises(ValueError): + padder(contents, None, np.int64) + contents = [[1, 2]] + # 不能是2维 + with self.assertRaises(ValueError): + padder(contents, None, np.int64) + contents = [[[[1, 2]]]] + # 不能是3维以上 + with self.assertRaises(ValueError): + padder(contents, None, np.int64) + + contents = [ + [[1, 2, 3], [4, 5], [7,8,9,10]], + [[1]] + ] + self.assertListEqual([[[1, 2, 3, 0], [4, 5, 0, 0], [7, 8, 9, 10]], [[1, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]], + padder(contents, None, np.int64).tolist()) + + padder = EngChar2DPadder(pad_length=5, pad_val=-100) + self.assertListEqual( + [[[1, 2, 3, -100, -100], [4, 5, -100, -100, -100], [7, 8, 9, 10, -100]], + [[1, -100, -100, -100, -100], [-100, -100, -100, -100, -100], [-100, -100, -100, -100, -100]]], + padder(contents, None, np.int64).tolist() + ) + + def test_None_dtype(self): + from fastNLP import AutoPadder + padder = AutoPadder() + content = [ + [[1, 2, 3], [4, 5], [7, 8, 9, 10]], + [[1]] + ] + ans = padder(content, None, None).tolist() + self.assertListEqual(content, ans) diff --git a/test/core/test_fieldarray.py b/test/core/test_fieldarray.py deleted file mode 100644 index 1204cda5..00000000 --- a/test/core/test_fieldarray.py +++ /dev/null @@ -1,99 +0,0 @@ -import unittest - -import numpy as np - -from fastNLP.core.fieldarray import FieldArray - - -class TestFieldArray(unittest.TestCase): - def test(self): - fa = FieldArray("x", [1, 2, 3, 4, 5], is_input=True) - self.assertEqual(len(fa), 5) - fa.append(6) - self.assertEqual(len(fa), 6) - - self.assertEqual(fa[-1], 6) - self.assertEqual(fa[0], 1) - fa[-1] = 60 - self.assertEqual(fa[-1], 60) - - self.assertEqual(fa.get(0), 1) - self.assertTrue(isinstance(fa.get([0, 1, 2]), np.ndarray)) - self.assertListEqual(list(fa.get([0, 1, 2])), [1, 2, 3]) - - def test_type_conversion(self): - fa = FieldArray("x", [1.2, 2.2, 3, 4, 5], is_input=True) - self.assertEqual(fa.pytype, float) - self.assertEqual(fa.dtype, np.float64) - - fa = FieldArray("x", [1, 2, 3, 4, 5], is_input=True) - fa.append(1.3333) - self.assertEqual(fa.pytype, float) - self.assertEqual(fa.dtype, np.float64) - - fa = FieldArray("y", [1.1, 2.2, 3.3, 4.4, 5.5], is_input=True) - fa.append(10) - self.assertEqual(fa.pytype, float) - self.assertEqual(fa.dtype, np.float64) - - fa = FieldArray("y", ["a", "b", "c", "d"], is_input=True) - fa.append("e") - self.assertEqual(fa.dtype, np.str) - self.assertEqual(fa.pytype, str) - - def test_support_np_array(self): - fa = FieldArray("y", [np.array([1.1, 2.2, 3.3, 4.4, 5.5])], is_input=True) - self.assertEqual(fa.dtype, np.ndarray) - self.assertEqual(fa.pytype, np.ndarray) - - fa.append(np.array([1.1, 2.2, 3.3, 4.4, 5.5])) - self.assertEqual(fa.dtype, np.ndarray) - self.assertEqual(fa.pytype, np.ndarray) - - fa = FieldArray("my_field", np.random.rand(3, 5), is_input=True) - # in this case, pytype is actually a float. We do not care about it. - self.assertEqual(fa.dtype, np.float64) - - def test_nested_list(self): - fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1.1, 2.2, 3.3, 4.4, 5.5]], is_input=True) - self.assertEqual(fa.pytype, float) - self.assertEqual(fa.dtype, np.float64) - - def test_getitem_v1(self): - fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) - self.assertEqual(fa[0], [1.1, 2.2, 3.3, 4.4, 5.5]) - ans = fa[[0, 1]] - self.assertTrue(isinstance(ans, np.ndarray)) - self.assertTrue(isinstance(ans[0], np.ndarray)) - self.assertEqual(ans[0].tolist(), [1.1, 2.2, 3.3, 4.4, 5.5]) - self.assertEqual(ans[1].tolist(), [1, 2, 3, 4, 5]) - self.assertEqual(ans.dtype, np.float64) - - def test_getitem_v2(self): - x = np.random.rand(10, 5) - fa = FieldArray("my_field", x, is_input=True) - indices = [0, 1, 3, 4, 6] - for a, b in zip(fa[indices], x[indices]): - self.assertListEqual(a.tolist(), b.tolist()) - - def test_append(self): - with self.assertRaises(Exception): - fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) - fa.append(0) - - with self.assertRaises(Exception): - fa = FieldArray("y", [1.1, 2.2, 3.3, 4.4, 5.5], is_input=True) - fa.append([1, 2, 3, 4, 5]) - - with self.assertRaises(Exception): - fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) - fa.append([]) - - with self.assertRaises(Exception): - fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) - fa.append(["str", 0, 0, 0, 1.89]) - - fa = FieldArray("y", [[1.1, 2.2, 3.3, 4.4, 5.5], [1, 2, 3, 4, 5]], is_input=True) - fa.append([1.2, 2.3, 3.4, 4.5, 5.6]) - self.assertEqual(len(fa), 3) - self.assertEqual(fa[2], [1.2, 2.3, 3.4, 4.5, 5.6]) diff --git a/test/core/test_instance.py b/test/core/test_instance.py index 1342ba2c..207b44e9 100644 --- a/test/core/test_instance.py +++ b/test/core/test_instance.py @@ -1,33 +1,33 @@ import unittest -from fastNLP.core.instance import Instance +from fastNLP import Instance class TestCase(unittest.TestCase): - + def test_init(self): fields = {"x": [1, 2, 3], "y": [4, 5, 6]} ins = Instance(x=[1, 2, 3], y=[4, 5, 6]) self.assertTrue(isinstance(ins.fields, dict)) self.assertEqual(ins.fields, fields) - + ins = Instance(**fields) self.assertEqual(ins.fields, fields) - + def test_add_field(self): fields = {"x": [1, 2, 3], "y": [4, 5, 6]} ins = Instance(**fields) ins.add_field("z", [1, 1, 1]) fields.update({"z": [1, 1, 1]}) self.assertEqual(ins.fields, fields) - + def test_get_item(self): fields = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [1, 1, 1]} ins = Instance(**fields) self.assertEqual(ins["x"], [1, 2, 3]) self.assertEqual(ins["y"], [4, 5, 6]) self.assertEqual(ins["z"], [1, 1, 1]) - + def test_repr(self): fields = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [1, 1, 1]} ins = Instance(**fields) diff --git a/test/core/test_loss.py b/test/core/test_loss.py index a6d542fa..8db54615 100644 --- a/test/core/test_loss.py +++ b/test/core/test_loss.py @@ -3,7 +3,7 @@ import unittest import torch import torch.nn.functional as F -import fastNLP.core.losses as loss +import fastNLP as loss from fastNLP.core.losses import squash, unpad @@ -14,21 +14,21 @@ class TestLoss(unittest.TestCase): b = torch.empty(3, dtype=torch.long).random_(5) ans = ce({"my_predict": a}, {"my_truth": b}) self.assertEqual(ans, torch.nn.functional.cross_entropy(a, b)) - + def test_BCELoss(self): bce = loss.BCELoss(pred="my_predict", target="my_truth") a = torch.sigmoid(torch.randn((3, 5), requires_grad=False)) b = torch.randn((3, 5), requires_grad=False) ans = bce({"my_predict": a}, {"my_truth": b}) self.assertEqual(ans, torch.nn.functional.binary_cross_entropy(a, b)) - + def test_L1Loss(self): l1 = loss.L1Loss(pred="my_predict", target="my_truth") a = torch.randn(3, 5, requires_grad=False) b = torch.randn(3, 5) ans = l1({"my_predict": a}, {"my_truth": b}) self.assertEqual(ans, torch.nn.functional.l1_loss(a, b)) - + def test_NLLLoss(self): l1 = loss.NLLLoss(pred="my_predict", target="my_truth") a = F.log_softmax(torch.randn(3, 5, requires_grad=False), dim=0) @@ -43,34 +43,34 @@ class TestLosserError(unittest.TestCase): pred_dict = {"pred": torch.zeros(4, 3)} target_dict = {'target': torch.zeros(4).long()} los = loss.CrossEntropyLoss() - + print(los(pred_dict=pred_dict, target_dict=target_dict)) - + # def test_losser2(self): # (2) with corrupted size pred_dict = {"pred": torch.zeros(16, 3)} target_dict = {'target': torch.zeros(16, 3).long()} los = loss.CrossEntropyLoss() - + with self.assertRaises(RuntimeError): print(los(pred_dict=pred_dict, target_dict=target_dict)) - + def test_losser3(self): # (2) with corrupted size pred_dict = {"pred": torch.zeros(16, 3), 'stop_fast_param': 0} target_dict = {'target': torch.zeros(16).long()} los = loss.CrossEntropyLoss() - + print(los(pred_dict=pred_dict, target_dict=target_dict)) - + def test_check_error(self): l1 = loss.NLLLoss(pred="my_predict", target="my_truth") a = F.log_softmax(torch.randn(3, 5, requires_grad=False), dim=0) b = torch.tensor([1, 0, 4]) with self.assertRaises(Exception): ans = l1({"wrong_predict": a, "my": b}, {"my_truth": b}) - + with self.assertRaises(Exception): ans = l1({"my_predict": a}, {"truth": b, "my": a}) @@ -80,7 +80,7 @@ class TestLossUtils(unittest.TestCase): a, b = squash(torch.randn(3, 5), torch.randn(3, 5)) self.assertEqual(tuple(a.size()), (3, 5)) self.assertEqual(tuple(b.size()), (15,)) - + def test_unpad(self): a, b = unpad(torch.randn(5, 8, 3), torch.randn(5, 8)) self.assertEqual(tuple(a.size()), (5, 8, 3)) diff --git a/test/core/test_metrics.py b/test/core/test_metrics.py index 80ed54e2..f3b0178c 100644 --- a/test/core/test_metrics.py +++ b/test/core/test_metrics.py @@ -3,10 +3,40 @@ import unittest import numpy as np import torch -from fastNLP.core.metrics import AccuracyMetric -from fastNLP.core.metrics import BMESF1PreRecMetric -from fastNLP.core.metrics import pred_topk, accuracy_topk - +from fastNLP import AccuracyMetric +from fastNLP.core.metrics import _pred_topk, _accuracy_topk +from fastNLP.core.vocabulary import Vocabulary +from collections import Counter +from fastNLP.core.metrics import SpanFPreRecMetric + + +def _generate_tags(encoding_type, number_labels=4): + vocab = {} + for i in range(number_labels): + label = str(i) + for tag in encoding_type: + if tag == 'O': + if tag not in vocab: + vocab['O'] = len(vocab) + 1 + continue + vocab['{}-{}'.format(tag, label)] = len(vocab) + 1 # 其实表达的是这个的count + return vocab + + +def _convert_res_to_fastnlp_res(metric_result): + allen_result = {} + key_map = {'f1-measure-overall': "f", "recall-overall": "rec", "precision-overall": "pre"} + for key, value in metric_result.items(): + if key in key_map: + key = key_map[key] + else: + label = key.split('-')[-1] + if key.startswith('f1'): + key = 'f-{}'.format(label) + else: + key = '{}-{}'.format(key[:3], label) + allen_result[key] = round(value, 6) + return allen_result class TestAccuracyMetric(unittest.TestCase): def test_AccuracyMetric1(self): @@ -14,24 +44,24 @@ class TestAccuracyMetric(unittest.TestCase): pred_dict = {"pred": torch.zeros(4, 3)} target_dict = {'target': torch.zeros(4)} metric = AccuracyMetric() - - metric(pred_dict=pred_dict, target_dict=target_dict, ) + + metric(pred_dict=pred_dict, target_dict=target_dict) print(metric.get_metric()) - + def test_AccuracyMetric2(self): # (2) with corrupted size try: pred_dict = {"pred": torch.zeros(4, 3, 2)} target_dict = {'target': torch.zeros(4)} metric = AccuracyMetric() - + metric(pred_dict=pred_dict, target_dict=target_dict, ) print(metric.get_metric()) except Exception as e: print(e) return - self.assertTrue(True, False), "No exception catches." - + print("No exception catches.") + def test_AccuracyMetric3(self): # (3) the second batch is corrupted size try: @@ -39,17 +69,17 @@ class TestAccuracyMetric(unittest.TestCase): pred_dict = {"pred": torch.zeros(4, 3, 2)} target_dict = {'target': torch.zeros(4, 3)} metric(pred_dict=pred_dict, target_dict=target_dict) - + pred_dict = {"pred": torch.zeros(4, 3, 2)} target_dict = {'target': torch.zeros(4)} metric(pred_dict=pred_dict, target_dict=target_dict) - + print(metric.get_metric()) except Exception as e: print(e) return self.assertTrue(True, False), "No exception catches." - + def test_AccuaryMetric4(self): # (5) check reset metric = AccuracyMetric() @@ -61,7 +91,7 @@ class TestAccuracyMetric(unittest.TestCase): self.assertTrue(isinstance(res, dict)) self.assertTrue("acc" in res) self.assertAlmostEqual(res["acc"], float(ans.float().mean()), places=3) - + def test_AccuaryMetric5(self): # (5) check reset metric = AccuracyMetric() @@ -71,7 +101,7 @@ class TestAccuracyMetric(unittest.TestCase): res = metric.get_metric(reset=False) ans = (torch.argmax(pred_dict["pred"], dim=2).float() == target_dict["target"]).float().mean() self.assertAlmostEqual(res["acc"], float(ans), places=4) - + def test_AccuaryMetric6(self): # (6) check numpy array is not acceptable try: @@ -83,7 +113,7 @@ class TestAccuracyMetric(unittest.TestCase): print(e) return self.assertTrue(True, False), "No exception catches." - + def test_AccuaryMetric7(self): # (7) check map, match metric = AccuracyMetric(pred='predictions', target='targets') @@ -93,12 +123,11 @@ class TestAccuracyMetric(unittest.TestCase): res = metric.get_metric() ans = (torch.argmax(pred_dict["predictions"], dim=2).float() == target_dict["targets"]).float().mean() self.assertAlmostEqual(res["acc"], float(ans), places=4) - + def test_AccuaryMetric8(self): - # (8) check map, does not match. use stop_fast_param to stop fast param map try: metric = AccuracyMetric(pred='predictions', target='targets') - pred_dict = {"prediction": torch.zeros(4, 3, 2), "stop_fast_param": 1} + pred_dict = {"prediction": torch.zeros(4, 3, 2)} target_dict = {'targets': torch.zeros(4, 3)} metric(pred_dict=pred_dict, target_dict=target_dict, ) self.assertDictEqual(metric.get_metric(), {'acc': 1}) @@ -106,7 +135,7 @@ class TestAccuracyMetric(unittest.TestCase): print(e) return self.assertTrue(True, False), "No exception catches." - + def test_AccuaryMetric9(self): # (9) check map, include unused try: @@ -119,12 +148,12 @@ class TestAccuracyMetric(unittest.TestCase): print(e) return self.assertTrue(True, False), "No exception catches." - + def test_AccuaryMetric10(self): # (10) check _fast_metric try: metric = AccuracyMetric() - pred_dict = {"predictions": torch.zeros(4, 3, 2), "masks": torch.zeros(4, 3)} + pred_dict = {"predictions": torch.zeros(4, 3, 2), "seq_len": torch.ones(3) * 3} target_dict = {'targets': torch.zeros(4, 3)} metric(pred_dict=pred_dict, target_dict=target_dict) self.assertDictEqual(metric.get_metric(), {'acc': 1}) @@ -132,242 +161,181 @@ class TestAccuracyMetric(unittest.TestCase): print(e) return self.assertTrue(True, False), "No exception catches." + + def test_seq_len(self): + N = 256 + seq_len = torch.zeros(N).long() + seq_len[0] = 2 + pred = {'pred': torch.ones(N, 2)} + target = {'target': torch.ones(N, 2), 'seq_len': seq_len} + metric = AccuracyMetric() + metric(pred_dict=pred, target_dict=target) + self.assertDictEqual(metric.get_metric(), {'acc': 1.}) + seq_len[1:] = 1 + metric(pred_dict=pred, target_dict=target) + self.assertDictEqual(metric.get_metric(), {'acc': 1.}) + class SpanF1PreRecMetric(unittest.TestCase): def test_case1(self): - from fastNLP.core.metrics import bmes_tag_to_spans - from fastNLP.core.metrics import bio_tag_to_spans - + from fastNLP.core.metrics import _bmes_tag_to_spans + from fastNLP.core.metrics import _bio_tag_to_spans + bmes_lst = ['M-8', 'S-2', 'S-0', 'B-9', 'B-6', 'E-5', 'B-7', 'S-2', 'E-7', 'S-8'] bio_lst = ['O-8', 'O-2', 'B-0', 'O-9', 'I-6', 'I-5', 'I-7', 'I-2', 'I-7', 'O-8'] expect_bmes_res = set() - expect_bmes_res.update([('8', (0, 0)), ('2', (1, 1)), ('0', (2, 2)), ('9', (3, 3)), ('6', (4, 4)), - ('5', (5, 5)), ('7', (6, 6)), ('2', (7, 7)), ('7', (8, 8)), ('8', (9, 9))]) + expect_bmes_res.update([('8', (0, 1)), ('2', (1, 2)), ('0', (2, 3)), ('9', (3, 4)), ('6', (4, 5)), + ('5', (5, 6)), ('7', (6, 7)), ('2', (7, 8)), ('7', (8, 9)), ('8', (9, 10))]) expect_bio_res = set() - expect_bio_res.update([('7', (8, 8)), ('0', (2, 2)), ('2', (7, 7)), ('5', (5, 5)), - ('6', (4, 4)), ('7', (6, 6))]) - self.assertSetEqual(expect_bmes_res,set(bmes_tag_to_spans(bmes_lst))) - self.assertSetEqual(expect_bio_res, set(bio_tag_to_spans(bio_lst))) - # 已与allennlp对应函数做过验证，但由于测试不能依赖allennlp，所以这里只是截取上面的例子做固定测试 - # from allennlp.data.dataset_readers.dataset_utils import bio_tags_to_spans as allen_bio_tags_to_spans - # from allennlp.data.dataset_readers.dataset_utils import bmes_tags_to_spans as allen_bmes_tags_to_spans - # for i in range(1000): - # strs = list(map(str, np.random.randint(100, size=1000))) - # bmes = list('bmes'.upper()) - # bmes_strs = [str_ + '-' + tag for tag, str_ in zip(strs, np.random.choice(bmes, size=len(strs)))] - # bio = list('bio'.upper()) - # bio_strs = [str_ + '-' + tag for tag, str_ in zip(strs, np.random.choice(bio, size=len(strs)))] - # self.assertSetEqual(set(allen_bmes_tags_to_spans(bmes_strs)),set(bmes_tag_to_spans(bmes_strs))) - # self.assertSetEqual(set(allen_bio_tags_to_spans(bio_strs)), set(bio_tag_to_spans(bio_strs))) + expect_bio_res.update([('7', (8, 9)), ('0', (2, 3)), ('2', (7, 8)), ('5', (5, 6)), + ('6', (4, 5)), ('7', (6, 7))]) + self.assertSetEqual(expect_bmes_res, set(_bmes_tag_to_spans(bmes_lst))) + self.assertSetEqual(expect_bio_res, set(_bio_tag_to_spans(bio_lst))) def test_case2(self): # 测试不带label的 - from fastNLP.core.metrics import bmes_tag_to_spans - from fastNLP.core.metrics import bio_tag_to_spans - + from fastNLP.core.metrics import _bmes_tag_to_spans + from fastNLP.core.metrics import _bio_tag_to_spans + bmes_lst = ['B', 'E', 'B', 'S', 'B', 'M', 'E', 'M', 'B', 'E'] bio_lst = ['I', 'B', 'O', 'O', 'I', 'O', 'I', 'B', 'O', 'O'] expect_bmes_res = set() - expect_bmes_res.update([('', (0, 1)), ('', (2, 2)), ('', (3, 3)), ('', (4, 6)), ('', (7, 7)), ('', (8, 9))]) + expect_bmes_res.update([('', (0, 2)), ('', (2, 3)), ('', (3, 4)), ('', (4, 7)), ('', (7, 8)), ('', (8, 10))]) expect_bio_res = set() - expect_bio_res.update([('', (7, 7)), ('', (6, 6)), ('', (4, 4)), ('', (0, 0)), ('', (1, 1))]) - self.assertSetEqual(expect_bmes_res,set(bmes_tag_to_spans(bmes_lst))) - self.assertSetEqual(expect_bio_res, set(bio_tag_to_spans(bio_lst))) - # 已与allennlp对应函数做过验证，但由于测试不能依赖allennlp，所以这里只是截取上面的例子做固定测试 - # from allennlp.data.dataset_readers.dataset_utils import bio_tags_to_spans as allen_bio_tags_to_spans - # from allennlp.data.dataset_readers.dataset_utils import bmes_tags_to_spans as allen_bmes_tags_to_spans - # for i in range(1000): - # bmes = list('bmes'.upper()) - # bmes_strs = np.random.choice(bmes, size=1000) - # bio = list('bio'.upper()) - # bio_strs = np.random.choice(bio, size=100) - # self.assertSetEqual(set(allen_bmes_tags_to_spans(bmes_strs)),set(bmes_tag_to_spans(bmes_strs))) - # self.assertSetEqual(set(allen_bio_tags_to_spans(bio_strs)), set(bio_tag_to_spans(bio_strs))) - - def tese_case3(self): - from fastNLP.core.vocabulary import Vocabulary - from collections import Counter - from fastNLP.core.metrics import SpanFPreRecMetric - # 与allennlp测试能否正确计算f metric - # - def generate_allen_tags(encoding_type, number_labels=4): - vocab = {} - for i in range(number_labels): - label = str(i) - for tag in encoding_type: - if tag == 'O': - if tag not in vocab: - vocab['O'] = len(vocab) + 1 - continue - vocab['{}-{}'.format(tag, label)] = len(vocab) + 1 # 其实表达的是这个的count - return vocab + expect_bio_res.update([('', (7, 8)), ('', (6, 7)), ('', (4, 5)), ('', (0, 1)), ('', (1, 2))]) + self.assertSetEqual(expect_bmes_res, set(_bmes_tag_to_spans(bmes_lst))) + self.assertSetEqual(expect_bio_res, set(_bio_tag_to_spans(bio_lst))) + def test_case3(self): number_labels = 4 # bio tag fastnlp_bio_vocab = Vocabulary(unknown=None, padding=None) - fastnlp_bio_vocab.word_count = Counter(generate_allen_tags('BIO', number_labels)) + fastnlp_bio_vocab.word_count = Counter(_generate_tags('BIO', number_labels)) fastnlp_bio_metric = SpanFPreRecMetric(tag_vocab=fastnlp_bio_vocab, only_gross=False) - bio_sequence = torch.FloatTensor( - [[[-0.9543, -1.4357, -0.2365, 0.2438, 1.0312, -1.4302, 0.3011, - 0.0470, 0.0971], - [-0.6638, -0.7116, -1.9804, 0.2787, -0.2732, -0.9501, -1.4523, - 0.7987, -0.3970], - [0.2939, 0.8132, -0.0903, -2.8296, 0.2080, -0.9823, -0.1898, - 0.6880, 1.4348], - [-0.1886, 0.0067, -0.6862, -0.4635, 2.2776, 0.0710, -1.6793, - -1.6876, -0.8917], - [-0.7663, 0.6377, 0.8669, 0.1237, 1.7628, 0.0313, -1.0824, - 1.4217, 0.2622]], + bio_sequence = torch.FloatTensor([[[-0.4424, -0.4579, -0.7376, 1.8129, 0.1316, 1.6566, -1.2169, + -0.3782, 0.8240], + [-1.2348, -0.1876, -0.1462, -0.4834, -0.6692, -0.9735, 1.1563, + -0.3562, -1.4116], + [ 1.6550, -0.9555, 0.3782, -1.3160, -1.5835, -0.3443, -1.7858, + 2.0023, 0.7075], + [-0.3772, -0.5447, -1.5631, 1.1614, 1.4598, -1.2764, 0.5186, + 0.3832, -0.1540], + [-0.1011, 0.0600, 1.1090, -0.3545, 0.1284, 1.1484, -1.0120, + -1.3508, -0.9513], + [ 1.8948, 0.8627, -2.1359, 1.3740, -0.7499, 1.5019, 0.6919, + -0.0842, -0.4294]], + + [[-0.2802, 0.6941, -0.4788, -0.3845, 1.7752, 1.2950, -1.9490, + -1.4138, -0.8853], + [-1.3752, -0.5457, -0.5305, 0.4018, 0.2934, 0.7931, 2.3845, + -1.0726, 0.0364], + [ 0.3621, 0.2609, 0.1269, -0.5950, 0.7212, 0.5959, 1.6264, + -0.8836, -0.9320], + [ 0.2003, -1.0758, -1.1560, -0.6472, -1.7549, 0.1264, 0.6044, + -1.6857, 1.1571], + [ 1.4277, -0.4915, 0.4496, 2.2027, 0.0730, -3.1792, -0.5125, + -0.5837, 1.0184], + [ 1.9495, 1.7145, -0.2143, -0.1230, -0.2205, 0.8250, 0.4943, + -0.9025, 0.0864]]]) + bio_target = torch.LongTensor([[3, 6, 0, 8, 2, 4], + [4, 1, 7, 0, 4, 7]]) + fastnlp_bio_metric({'pred': bio_sequence, 'seq_len': torch.LongTensor([6, 6])}, {'target': bio_target}) + expect_bio_res = {'pre-1': 0.333333, 'rec-1': 0.333333, 'f-1': 0.333333, 'pre-2': 0.5, 'rec-2': 0.5, + 'f-2': 0.5, 'pre-0': 0.0, 'rec-0': 0.0, 'f-0': 0.0, 'pre-3': 0.0, 'rec-3': 0.0, + 'f-3': 0.0, 'pre': 0.222222, 'rec': 0.181818, 'f': 0.2} - [[0.1529, 0.7474, -0.9037, 1.5287, 0.2771, 0.2223, 0.8136, - 1.3592, -0.8973], - [0.4515, -0.5235, 0.3265, -1.1947, 0.8308, 1.8754, -0.4887, - -0.4025, -0.3417], - [-0.7855, 0.1615, -0.1272, -1.9289, -0.5181, 1.9742, -0.9698, - 0.2861, -0.3966], - [-0.8291, -0.8823, -1.1496, 0.2164, 1.3390, -0.3964, -0.5275, - 0.0213, 1.4777], - [-1.1299, 0.0627, -0.1358, -1.5951, 0.4484, -0.6081, -1.9566, - 1.3024, 0.2001]]] - ) - bio_target = torch.LongTensor([[5., 0., 3., 3., 3.], - [5., 6., 8., 6., 0.]]) - fastnlp_bio_metric({'pred': bio_sequence, 'seq_lens': torch.LongTensor([5, 5])}, {'target': bio_target}) - expect_bio_res = {'pre-1': 0.24999999999999373, 'rec-1': 0.499999999999975, 'f-1': 0.33333333333327775, - 'pre-2': 0.0, 'rec-2': 0.0, 'f-2': 0.0, 'pre-3': 0.0, 'rec-3': 0.0, 'f-3': 0.0, 'pre-0': 0.0, - 'rec-0': 0.0, 'f-0': 0.0, 'pre': 0.12499999999999845, 'rec': 0.12499999999999845, - 'f': 0.12499999999994846} self.assertDictEqual(expect_bio_res, fastnlp_bio_metric.get_metric()) - #bmes tag - bmes_sequence = torch.FloatTensor( - [[[0.6536, -0.7179, 0.6579, 1.2503, 0.4176, 0.6696, 0.2352, - -0.4085, 0.4084, -0.4185, 1.4172, -0.9162, -0.2679, 0.3332, - -0.3505, -0.6002], - [0.3238, -1.2378, -1.3304, -0.4903, 1.4518, -0.1868, -0.7641, - 1.6199, -0.8877, 0.1449, 0.8995, -0.5810, 0.1041, 0.1002, - 0.4439, 0.2514], - [-0.8362, 2.9526, 0.8008, 0.1193, 1.0488, 0.6670, 1.1696, - -1.1006, -0.8540, -0.1600, -0.9519, -0.2749, -0.4948, -1.4753, - 0.5802, -0.0516], - [-0.8383, -1.7292, -1.4079, -1.5023, 0.5383, 0.6653, 0.3121, - 4.1249, -0.4173, -0.2043, 1.7755, 1.1110, -1.7069, -0.0390, - -0.9242, -0.0333], - [0.9088, -0.4955, -0.5076, 0.3732, 0.0283, -0.0263, -1.0393, - 0.7734, 1.0968, 0.4132, -1.3647, -0.5762, 0.6678, 0.8809, - -0.3779, -0.3195]], - - [[-0.4638, -0.5939, -0.1052, -0.5573, 0.4600, -1.3484, 0.1753, - 0.0685, 0.3663, -0.6789, 0.0097, 1.0327, -0.0212, -0.9957, - -0.1103, 0.4417], - [-0.2903, 0.9205, -1.5758, -1.0421, 0.2921, -0.2142, -0.3049, - -0.0879, -0.4412, -1.3195, -0.0657, -0.2986, 0.7214, 0.0631, - -0.6386, 0.2797], - [0.6440, -0.3748, 1.2912, -0.0170, 0.7447, 1.4075, -0.4947, - 0.4123, -0.8447, -0.5502, 0.3520, -0.2832, 0.5019, -0.1522, - 1.1237, -1.5385], - [0.2839, -0.7649, 0.9067, -0.1163, -1.3789, 0.2571, -1.3977, - -0.3680, -0.8902, -0.6983, -1.1583, 1.2779, 0.2197, 0.1376, - -0.0591, -0.2461], - [-0.2977, -1.8564, -0.5347, 1.0011, -1.1260, 0.4252, -2.0097, - 2.6973, -0.8308, -1.4939, 0.9865, -0.3935, 0.2743, 0.1142, - -0.7344, -1.2046]]] - ) - bmes_target = torch.LongTensor([[ 9., 6., 1., 9., 15.], - [ 6., 15., 6., 15., 5.]]) - - fastnlp_bmes_vocab = Vocabulary(unknown=None, padding=None) - fastnlp_bmes_vocab.word_count = Counter(generate_allen_tags('BMES', number_labels)) - fastnlp_bmes_metric = SpanFPreRecMetric(tag_vocab=fastnlp_bmes_vocab, only_gross=False, encoding_type='bmes') - fastnlp_bmes_metric({'pred': bmes_sequence, 'seq_lens': torch.LongTensor([20, 20])}, {'target': bmes_target}) - - expect_bmes_res = {'f-3': 0.6666666666665778, 'pre-3': 0.499999999999975, 'rec-3': 0.9999999999999001, - 'f-0': 0.0, 'pre-0': 0.0, 'rec-0': 0.0, 'f-1': 0.33333333333327775, - 'pre-1': 0.24999999999999373, 'rec-1': 0.499999999999975, 'f-2': 0.7499999999999314, - 'pre-2': 0.7499999999999812, 'rec-2': 0.7499999999999812, 'f': 0.49999999999994504, - 'pre': 0.499999999999995, 'rec': 0.499999999999995} - - self.assertDictEqual(fastnlp_bmes_metric.get_metric(), expect_bmes_res) - - # 已经和allennlp做过验证，但由于不能依赖allennlp，所以注释了以下代码 - # from allennlp.data.vocabulary import Vocabulary as allen_Vocabulary - # from allennlp.training.metrics import SpanBasedF1Measure - # allen_bio_vocab = allen_Vocabulary({"tags": generate_allen_tags('BIO', number_labels)}, - # non_padded_namespaces=['tags']) - # allen_bio_metric = SpanBasedF1Measure(allen_bio_vocab, 'tags') - # bio_sequence = torch.randn(size=(2, 20, 2 * number_labels + 1)) - # bio_target = torch.randint(2 * number_labels + 1, size=(2, 20)) - # allen_bio_metric(bio_sequence, bio_target, torch.ones(2, 20)) - # fastnlp_bio_vocab = Vocabulary(unknown=None, padding=None) - # fastnlp_bio_vocab.word_count = Counter(generate_allen_tags('BIO', number_labels)) - # fastnlp_bio_metric = SpanFPreRecMetric(tag_vocab=fastnlp_bio_vocab, only_gross=False) - # - # def convert_allen_res_to_fastnlp_res(metric_result): - # allen_result = {} - # key_map = {'f1-measure-overall': "f", "recall-overall": "rec", "precision-overall": "pre"} - # for key, value in metric_result.items(): - # if key in key_map: - # key = key_map[key] - # else: - # label = key.split('-')[-1] - # if key.startswith('f1'): - # key = 'f-{}'.format(label) - # else: - # key = '{}-{}'.format(key[:3], label) - # allen_result[key] = value - # return allen_result - # - # # print(convert_allen_res_to_fastnlp_res(allen_bio_metric.get_metric())) - # # print(fastnlp_bio_metric.get_metric()) - # self.assertDictEqual(convert_allen_res_to_fastnlp_res(allen_bio_metric.get_metric()), - # fastnlp_bio_metric.get_metric()) - # - # allen_bmes_vocab = allen_Vocabulary({"tags": generate_allen_tags('BMES', number_labels)}) - # allen_bmes_metric = SpanBasedF1Measure(allen_bmes_vocab, 'tags', label_encoding='BMES') - # fastnlp_bmes_vocab = Vocabulary(unknown=None, padding=None) - # fastnlp_bmes_vocab.word_count = Counter(generate_allen_tags('BMES', number_labels)) - # fastnlp_bmes_metric = SpanFPreRecMetric(tag_vocab=fastnlp_bmes_vocab, only_gross=False, encoding_type='bmes') - # bmes_sequence = torch.randn(size=(2, 20, 4 * number_labels)) - # bmes_target = torch.randint(4 * number_labels, size=(2, 20)) - # allen_bmes_metric(bmes_sequence, bmes_target, torch.ones(2, 20)) - # fastnlp_bmes_metric({'pred': bmes_sequence, 'seq_lens': torch.LongTensor([20, 20])}, {'target': bmes_target}) - # - # # print(convert_allen_res_to_fastnlp_res(allen_bmes_metric.get_metric())) - # # print(fastnlp_bmes_metric.get_metric()) - # self.assertDictEqual(convert_allen_res_to_fastnlp_res(allen_bmes_metric.get_metric()), - # fastnlp_bmes_metric.get_metric()) - -class TestBMESF1PreRecMetric(unittest.TestCase): - def test_case1(self): - seq_lens = torch.LongTensor([4, 2]) - pred = torch.randn(2, 4, 4) - target = torch.LongTensor([[0, 1, 2, 3], - [3, 3, 0, 0]]) - pred_dict = {'pred': pred} - target_dict = {'target': target, 'seq_lens': seq_lens} - - metric = BMESF1PreRecMetric() - metric(pred_dict, target_dict) - metric.get_metric() - - def test_case2(self): - # 测试相同两个seqence，应该给出{f1: 1, precision:1, recall:1} - seq_lens = torch.LongTensor([4, 2]) - target = torch.LongTensor([[0, 1, 2, 3], - [3, 3, 0, 0]]) - pred_dict = {'pred': target} - target_dict = {'target': target, 'seq_lens': seq_lens} - - metric = BMESF1PreRecMetric() - metric(pred_dict, target_dict) - self.assertDictEqual(metric.get_metric(), {'f': 1.0, 'pre': 1.0, 'rec': 1.0}) + def test_case4(self): + # bmes tag + def _generate_samples(): + target = [] + seq_len = [] + vocab = Vocabulary(unknown=None, padding=None) + for i in range(3): + target_i = [] + seq_len_i = 0 + for j in range(1, 10): + word_len = np.random.randint(1, 5) + seq_len_i += word_len + if word_len==1: + target_i.append('S') + else: + target_i.append('B') + target_i.extend(['M']*(word_len-2)) + target_i.append('E') + vocab.add_word_lst(target_i) + target.append(target_i) + seq_len.append(seq_len_i) + target_ = np.zeros((3, max(seq_len))) + for i in range(3): + target_i = [vocab.to_index(t) for t in target[i]] + target_[i, :seq_len[i]] = target_i + return target_, target, seq_len, vocab + def get_eval(raw_target, pred, vocab, seq_len): + pred = pred.argmax(dim=-1).tolist() + tp = 0 + gold = 0 + seg = 0 + pred_target = [] + for i in range(len(seq_len)): + tags = [vocab.to_word(p) for p in pred[i][:seq_len[i]]] + spans = [] + prev_bmes_tag = None + for idx, tag in enumerate(tags): + if tag in ('B', 'S'): + spans.append([idx, idx]) + elif tag in ('M', 'E') and prev_bmes_tag in ('B', 'M'): + spans[-1][1] = idx + else: + spans.append([idx, idx]) + prev_bmes_tag = tag + tmp = [] + for span in spans: + if span[1]-span[0]>0: + tmp.extend(['B'] + ['M']*(span[1]-span[0]-1) + ['E']) + else: + tmp.append('S') + pred_target.append(tmp) + for i in range(len(seq_len)): + raw_pred = pred_target[i] + start = 0 + for j in range(seq_len[i]): + if raw_target[i][j] in ('E', 'S'): + flag = True + for k in range(start, j+1): + if raw_target[i][k]!=raw_pred[k]: + flag = False + break + if flag: + tp += 1 + start = j + 1 + gold += 1 + if raw_pred[j] in ('E', 'S'): + seg += 1 + + pre = round(tp/seg, 6) + rec = round(tp/gold, 6) + return {'f': round(2*pre*rec/(pre+rec), 6), 'pre': pre, 'rec':rec} + + target, raw_target, seq_len, vocab = _generate_samples() + pred = torch.randn(3, max(seq_len), 4) + + expected_metric = get_eval(raw_target, pred, vocab, seq_len) + metric = SpanFPreRecMetric(vocab, encoding_type='bmes') + metric({'pred': pred, 'seq_len':torch.LongTensor(seq_len)}, {'target': torch.from_numpy(target)}) + # print(metric.get_metric(reset=False)) + # print(expected_metric) + metric_value = metric.get_metric() + for key, value in expected_metric.items(): + self.assertAlmostEqual(value, metric_value[key], places=5) class TestUsefulFunctions(unittest.TestCase): # 测试metrics.py中一些看上去挺有用的函数 def test_case_1(self): # multi-class - _ = accuracy_topk(np.random.randint(0, 3, size=(10, 1)), np.random.randint(0, 3, size=(10, 1)), k=3) - _ = pred_topk(np.random.randint(0, 3, size=(10, 1))) - + _ = _accuracy_topk(np.random.randint(0, 3, size=(10, 1)), np.random.randint(0, 3, size=(10, 1)), k=3) + _ = _pred_topk(np.random.randint(0, 3, size=(10, 1))) + # 跑通即可 diff --git a/test/core/test_optimizer.py b/test/core/test_optimizer.py index 83ed6000..b9a1c271 100644 --- a/test/core/test_optimizer.py +++ b/test/core/test_optimizer.py @@ -2,7 +2,7 @@ import unittest import torch -from fastNLP.core.optimizer import SGD, Adam +from fastNLP import SGD, Adam class TestOptim(unittest.TestCase): @@ -12,42 +12,42 @@ class TestOptim(unittest.TestCase): self.assertTrue("momentum" in optim.__dict__["settings"]) res = optim.construct_from_pytorch(torch.nn.Linear(10, 3).parameters()) self.assertTrue(isinstance(res, torch.optim.SGD)) - + optim = SGD(lr=0.001) self.assertEqual(optim.__dict__["settings"]["lr"], 0.001) res = optim.construct_from_pytorch(torch.nn.Linear(10, 3).parameters()) self.assertTrue(isinstance(res, torch.optim.SGD)) - + optim = SGD(lr=0.002, momentum=0.989) self.assertEqual(optim.__dict__["settings"]["lr"], 0.002) self.assertEqual(optim.__dict__["settings"]["momentum"], 0.989) - + optim = SGD(0.001) self.assertEqual(optim.__dict__["settings"]["lr"], 0.001) res = optim.construct_from_pytorch(torch.nn.Linear(10, 3).parameters()) self.assertTrue(isinstance(res, torch.optim.SGD)) - + with self.assertRaises(TypeError): _ = SGD("???") with self.assertRaises(TypeError): _ = SGD(0.001, lr=0.002) - + def test_Adam(self): optim = Adam(model_params=torch.nn.Linear(10, 3).parameters()) self.assertTrue("lr" in optim.__dict__["settings"]) self.assertTrue("weight_decay" in optim.__dict__["settings"]) res = optim.construct_from_pytorch(torch.nn.Linear(10, 3).parameters()) self.assertTrue(isinstance(res, torch.optim.Adam)) - + optim = Adam(lr=0.001) self.assertEqual(optim.__dict__["settings"]["lr"], 0.001) res = optim.construct_from_pytorch(torch.nn.Linear(10, 3).parameters()) self.assertTrue(isinstance(res, torch.optim.Adam)) - + optim = Adam(lr=0.002, weight_decay=0.989) self.assertEqual(optim.__dict__["settings"]["lr"], 0.002) self.assertEqual(optim.__dict__["settings"]["weight_decay"], 0.989) - + optim = Adam(0.001) self.assertEqual(optim.__dict__["settings"]["lr"], 0.001) res = optim.construct_from_pytorch(torch.nn.Linear(10, 3).parameters()) diff --git a/test/core/test_predictor.py b/test/core/test_predictor.py index 8be5f289..701353dc 100644 --- a/test/core/test_predictor.py +++ b/test/core/test_predictor.py @@ -1,4 +1,5 @@ import unittest +from collections import defaultdict import numpy as np import torch @@ -6,7 +7,6 @@ import torch from fastNLP.core.dataset import DataSet from fastNLP.core.instance import Instance from fastNLP.core.predictor import Predictor -from fastNLP.modules.encoder.linear import Linear def prepare_fake_dataset(): @@ -23,12 +23,26 @@ def prepare_fake_dataset(): return data_set +class LinearModel(torch.nn.Module): + def __init__(self): + super(LinearModel, self).__init__() + self.linear = torch.nn.Linear(2, 1) + + def forward(self, x): + return {"predict": self.linear(x)} + + class TestPredictor(unittest.TestCase): - def test(self): - predictor = Predictor() - model = Linear(2, 1) + def test_simple(self): + model = LinearModel() + predictor = Predictor(model) data = prepare_fake_dataset() data.set_input("x") - ans = predictor.predict(model, data) - self.assertEqual(len(ans), 2000) - self.assertTrue(isinstance(ans[0], torch.Tensor)) + ans = predictor.predict(data) + self.assertTrue(isinstance(ans, defaultdict)) + self.assertTrue("predict" in ans) + self.assertTrue(isinstance(ans["predict"], list)) + + def test_sequence(self): + # test sequence input/output + pass diff --git a/test/core/test_sampler.py b/test/core/test_sampler.py index b23af470..703a9428 100644 --- a/test/core/test_sampler.py +++ b/test/core/test_sampler.py @@ -3,18 +3,12 @@ import unittest import torch -from fastNLP.core.dataset import DataSet -from fastNLP.core.sampler import convert_to_torch_tensor, SequentialSampler, RandomSampler, \ - k_means_1d, k_means_bucketing, simple_sort_bucketing, BucketSampler +from fastNLP import DataSet +from fastNLP import SequentialSampler, RandomSampler, BucketSampler +from fastNLP.core.sampler import k_means_1d, k_means_bucketing, simple_sort_bucketing class TestSampler(unittest.TestCase): - def test_convert_to_torch_tensor(self): - data = [[1, 2, 3, 4, 5], [5, 4, 3, 2, 1], [1, 3, 4, 5, 2]] - ans = convert_to_torch_tensor(data, False) - assert isinstance(ans, torch.Tensor) - assert tuple(ans.shape) == (3, 5) - def test_sequential_sampler(self): sampler = SequentialSampler() data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 10] @@ -44,7 +38,7 @@ class TestSampler(unittest.TestCase): assert len(_) == 10 def test_BucketSampler(self): - sampler = BucketSampler(num_buckets=3, batch_size=16, seq_lens_field_name="seq_len") + sampler = BucketSampler(num_buckets=3, batch_size=16, seq_len_field_name="seq_len") data_set = DataSet({"x": [[0] * random.randint(1, 10)] * 10, "y": [[5, 6]] * 10}) data_set.apply(lambda ins: len(ins["x"]), new_field_name="seq_len") indices = sampler(data_set) diff --git a/test/core/test_tester.py b/test/core/test_tester.py index d606c0b8..d0267cce 100644 --- a/test/core/test_tester.py +++ b/test/core/test_tester.py @@ -1,32 +1,25 @@ import unittest +import numpy as np +from torch import nn +import time +from fastNLP import DataSet +from fastNLP import Instance +from fastNLP import AccuracyMetric +from fastNLP import Tester data_name = "pku_training.utf8" pickle_path = "data_for_tests" -import numpy as np -import torch.nn.functional as F -from torch import nn -import time -from fastNLP.core.utils import CheckError -from fastNLP.core.dataset import DataSet -from fastNLP.core.instance import Instance -from fastNLP.core.losses import BCELoss -from fastNLP.core.losses import CrossEntropyLoss -from fastNLP.core.metrics import AccuracyMetric -from fastNLP.core.optimizer import SGD -from fastNLP.core.tester import Tester -from fastNLP.models.base_model import NaiveClassifier - def prepare_fake_dataset(): mean = np.array([-3, -3]) cov = np.array([[1, 0], [0, 1]]) class_A = np.random.multivariate_normal(mean, cov, size=(1000,)) - + mean = np.array([3, 3]) cov = np.array([[1, 0], [0, 1]]) class_B = np.random.multivariate_normal(mean, cov, size=(1000,)) - + data_set = DataSet([Instance(x=[float(item[0]), float(item[1])], y=[0.0]) for item in class_A] + [Instance(x=[float(item[0]), float(item[1])], y=[1.0]) for item in class_B]) return data_set @@ -39,6 +32,7 @@ def prepare_fake_dataset2(*args, size=100): data[arg] = np.random.randn(size, 5) return DataSet(data=data) + class TestTester(unittest.TestCase): def test_case_1(self): # 检查报错提示能否正确提醒用户 @@ -46,10 +40,12 @@ class TestTester(unittest.TestCase): dataset.rename_field('x_unused', 'x2') dataset.set_input('x1', 'x2') dataset.set_target('y', 'x1') + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) + def forward(self, x1, x2): x1 = self.fc(x1) x2 = self.fc(x2) @@ -57,7 +53,7 @@ class TestTester(unittest.TestCase): time.sleep(0.1) # loss = F.cross_entropy(x, y) return {'preds': x} - + model = Model() with self.assertRaises(NameError): tester = Tester( diff --git a/test/core/test_trainer.py b/test/core/test_trainer.py index 624f2587..f559eac5 100644 --- a/test/core/test_trainer.py +++ b/test/core/test_trainer.py @@ -5,25 +5,24 @@ import numpy as np import torch.nn.functional as F from torch import nn -from fastNLP.core.dataset import DataSet -from fastNLP.core.instance import Instance -from fastNLP.core.losses import BCELoss -from fastNLP.core.losses import CrossEntropyLoss -from fastNLP.core.metrics import AccuracyMetric -from fastNLP.core.optimizer import SGD -from fastNLP.core.trainer import Trainer +from fastNLP import DataSet +from fastNLP import Instance +from fastNLP import BCELoss +from fastNLP import CrossEntropyLoss +from fastNLP import AccuracyMetric +from fastNLP import SGD +from fastNLP import Trainer from fastNLP.models.base_model import NaiveClassifier - def prepare_fake_dataset(): mean = np.array([-3, -3]) cov = np.array([[1, 0], [0, 1]]) class_A = np.random.multivariate_normal(mean, cov, size=(1000,)) - + mean = np.array([3, 3]) cov = np.array([[1, 0], [0, 1]]) class_B = np.random.multivariate_normal(mean, cov, size=(1000,)) - + data_set = DataSet([Instance(x=[float(item[0]), float(item[1])], y=[0.0]) for item in class_A] + [Instance(x=[float(item[0]), float(item[1])], y=[1.0]) for item in class_B]) return data_set @@ -42,11 +41,11 @@ class TrainerTestGround(unittest.TestCase): data_set = prepare_fake_dataset() data_set.set_input("x", flag=True) data_set.set_target("y", flag=True) - + train_set, dev_set = data_set.split(0.3) - + model = NaiveClassifier(2, 1) - + trainer = Trainer(train_set, model, loss=BCELoss(pred="predict", target="y"), metrics=AccuracyMetric(pred="predict", target="y"), @@ -63,26 +62,26 @@ class TrainerTestGround(unittest.TestCase): """ # 应该正确运行 """ - + def test_trainer_suggestion1(self): # 检查报错提示能否正确提醒用户。 # 这里没有传入forward需要的数据。需要trainer提醒用户如何设置。 dataset = prepare_fake_dataset2('x') - + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) - + def forward(self, x1, x2, y): x1 = self.fc(x1) x2 = self.fc(x2) x = x1 + x2 loss = F.cross_entropy(x, y) return {'loss': loss} - + model = Model() - + with self.assertRaises(RuntimeError): trainer = Trainer( train_data=dataset, @@ -97,25 +96,25 @@ class TrainerTestGround(unittest.TestCase): (2). You need to provide ['x1', 'x2'] in DataSet and set it as input. """ - + def test_trainer_suggestion2(self): # 检查报错提示能否正确提醒用户 # 这里传入forward需要的数据，看是否可以运行 dataset = prepare_fake_dataset2('x1', 'x2') dataset.set_input('x1', 'x2', 'y', flag=True) - + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) - + def forward(self, x1, x2, y): x1 = self.fc(x1) x2 = self.fc(x2) x = x1 + x2 loss = F.cross_entropy(x, y) return {'loss': loss} - + model = Model() trainer = Trainer( train_data=dataset, @@ -127,25 +126,25 @@ class TrainerTestGround(unittest.TestCase): """ # 应该正确运行 """ - + def test_trainer_suggestion3(self): # 检查报错提示能否正确提醒用户 # 这里传入forward需要的数据，但是forward没有返回loss这个key dataset = prepare_fake_dataset2('x1', 'x2') dataset.set_input('x1', 'x2', 'y', flag=True) - + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) - + def forward(self, x1, x2, y): x1 = self.fc(x1) x2 = self.fc(x2) x = x1 + x2 loss = F.cross_entropy(x, y) return {'wrong_loss_key': loss} - + model = Model() with self.assertRaises(NameError): trainer = Trainer( @@ -155,23 +154,25 @@ class TrainerTestGround(unittest.TestCase): print_every=2 ) trainer.train() - + def test_trainer_suggestion4(self): # 检查报错提示能否正确提醒用户 # 这里传入forward需要的数据，是否可以正确提示unused dataset = prepare_fake_dataset2('x1', 'x2') dataset.set_input('x1', 'x2', 'y', flag=True) + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) + def forward(self, x1, x2, y): x1 = self.fc(x1) x2 = self.fc(x2) x = x1 + x2 loss = F.cross_entropy(x, y) return {'losses': loss} - + model = Model() with self.assertRaises(NameError): trainer = Trainer( @@ -180,7 +181,7 @@ class TrainerTestGround(unittest.TestCase): use_tqdm=False, print_every=2 ) - + def test_trainer_suggestion5(self): # 检查报错提示能否正确提醒用户 # 这里传入多余参数，让其duplicate, 但这里因为y不会被调用，所以其实不会报错 @@ -188,17 +189,19 @@ class TrainerTestGround(unittest.TestCase): dataset.rename_field('x_unused', 'x2') dataset.set_input('x1', 'x2', 'y') dataset.set_target('y') + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) + def forward(self, x1, x2, y): x1 = self.fc(x1) x2 = self.fc(x2) x = x1 + x2 loss = F.cross_entropy(x, y) return {'loss': loss} - + model = Model() trainer = Trainer( train_data=dataset, @@ -206,7 +209,7 @@ class TrainerTestGround(unittest.TestCase): use_tqdm=False, print_every=2 ) - + def test_trainer_suggestion6(self): # 检查报错提示能否正确提醒用户 # 这里传入多余参数，让其duplicate @@ -214,10 +217,12 @@ class TrainerTestGround(unittest.TestCase): dataset.rename_field('x_unused', 'x2') dataset.set_input('x1', 'x2') dataset.set_target('y', 'x1') + class Model(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(5, 4) + def forward(self, x1, x2): x1 = self.fc(x1) x2 = self.fc(x2) @@ -225,7 +230,7 @@ class TrainerTestGround(unittest.TestCase): time.sleep(0.1) # loss = F.cross_entropy(x, y) return {'preds': x} - + model = Model() with self.assertRaises(NameError): trainer = Trainer( @@ -236,7 +241,33 @@ class TrainerTestGround(unittest.TestCase): metrics=AccuracyMetric(), use_tqdm=False, print_every=2) + + """ + def test_trainer_multiprocess(self): + dataset = prepare_fake_dataset2('x1', 'x2') + dataset.set_input('x1', 'x2', 'y', flag=True) + + class Model(nn.Module): + def __init__(self): + super().__init__() + self.fc = nn.Linear(5, 4) + + def forward(self, x1, x2, y): + x1 = self.fc(x1) + x2 = self.fc(x2) + x = x1 + x2 + loss = F.cross_entropy(x, y) + return {'loss': loss} - def test_case2(self): - # check metrics Wrong - data_set = prepare_fake_dataset2('x1', 'x2') + model = Model() + trainer = Trainer( + train_data=dataset, + model=model, + use_tqdm=True, + print_every=2, + num_workers=2, + pin_memory=False, + timeout=0, + ) + trainer.train() + """ diff --git a/test/core/test_utils.py b/test/core/test_utils.py new file mode 100644 index 00000000..91b5d00f --- /dev/null +++ b/test/core/test_utils.py @@ -0,0 +1,252 @@ +import unittest +import _pickle +from fastNLP import cache_results +from fastNLP.io import EmbedLoader +from fastNLP import DataSet +from fastNLP import Instance +import time +import os +import torch +from torch import nn +from fastNLP.core.utils import _move_model_to_device, _get_model_device +import numpy as np +from fastNLP.core.utils import seq_len_to_mask + +class Model(nn.Module): + def __init__(self): + super().__init__() + self.param = nn.Parameter(torch.zeros(0)) + + +class TestMoveModelDeivce(unittest.TestCase): + def test_case1(self): + # 测试str + model = Model() + model = _move_model_to_device(model, 'cpu') + assert model.param.device == torch.device('cpu') + # 测试不存在的device报错 + with self.assertRaises(Exception): + _move_model_to_device(model, 'cpuu') + # 测试gpu + if torch.cuda.is_available(): + model = _move_model_to_device(model, 'cuda') + assert model.param.is_cuda + model = _move_model_to_device(model, 'cuda:0') + assert model.param.device == torch.device('cuda:0') + with self.assertRaises(Exception): + _move_model_to_device(model, 'cuda:1000') + # 测试None + model = _move_model_to_device(model, None) + + def test_case2(self): + # 测试使用int初始化 + model = Model() + if torch.cuda.is_available(): + model = _move_model_to_device(model, 0) + assert model.param.device == torch.device('cuda:0') + assert model.param.device == torch.device('cuda:0'), "The model should be in " + with self.assertRaises(Exception): + _move_model_to_device(model, 100) + with self.assertRaises(Exception): + _move_model_to_device(model, -1) + + def test_case3(self): + # 测试None + model = Model() + device = _get_model_device(model) + model = _move_model_to_device(model, None) + assert device == _get_model_device(model), "The device should not change." + if torch.cuda.is_available(): + model.cuda() + device = _get_model_device(model) + model = _move_model_to_device(model, None) + assert device == _get_model_device(model), "The device should not change." + + model = nn.DataParallel(model, device_ids=[0]) + _move_model_to_device(model, None) + with self.assertRaises(Exception): + _move_model_to_device(model, 'cpu') + + def test_case4(self): + # 测试传入list的内容 + model = Model() + device = ['cpu'] + with self.assertRaises(Exception): + _move_model_to_device(model, device) + if torch.cuda.is_available(): + device = [0] + _model = _move_model_to_device(model, device) + assert not isinstance(_model, nn.DataParallel) + device = [torch.device('cuda:0'), torch.device('cuda:0')] + with self.assertRaises(Exception): + _model = _move_model_to_device(model, device) + if torch.cuda.device_count() > 1: + device = [0, 1] + _model = _move_model_to_device(model, device) + assert isinstance(_model, nn.DataParallel) + device = ['cuda', 'cuda:1'] + with self.assertRaises(Exception): + _move_model_to_device(model, device) + + def test_case5(self): + if not torch.cuda.is_available(): + return + # torch.device() + device = torch.device('cpu') + model = Model() + _move_model_to_device(model, device) + device = torch.device('cuda') + model = _move_model_to_device(model, device) + assert model.param.device == torch.device('cuda:0') + with self.assertRaises(Exception): + _move_model_to_device(model, torch.device('cuda:100')) + + +@cache_results('test/demo1.pkl') +def process_data_1(embed_file, cws_train): + embed, vocab = EmbedLoader.load_without_vocab(embed_file) + time.sleep(1) # 测试是否通过读取cache获得结果 + with open(cws_train, 'r', encoding='utf-8') as f: + d = DataSet() + for line in f: + line = line.strip() + if len(line) > 0: + d.append(Instance(raw=line)) + return embed, vocab, d + + +class TestCache(unittest.TestCase): + def test_cache_save(self): + try: + start_time = time.time() + embed, vocab, d = process_data_1('test/data_for_tests/word2vec_test.txt', 'test/data_for_tests/cws_train') + end_time = time.time() + pre_time = end_time - start_time + with open('test/demo1.pkl', 'rb') as f: + _embed, _vocab, _d = _pickle.load(f) + self.assertEqual(embed.shape, _embed.shape) + for i in range(embed.shape[0]): + self.assertListEqual(embed[i].tolist(), _embed[i].tolist()) + start_time = time.time() + embed, vocab, d = process_data_1('test/data_for_tests/word2vec_test.txt', 'test/data_for_tests/cws_train') + end_time = time.time() + read_time = end_time - start_time + print("Read using {:.3f}, while prepare using:{:.3f}".format(read_time, pre_time)) + self.assertGreater(pre_time - 0.5, read_time) + finally: + os.remove('test/demo1.pkl') + + def test_cache_save_overwrite_path(self): + try: + start_time = time.time() + embed, vocab, d = process_data_1('test/data_for_tests/word2vec_test.txt', 'test/data_for_tests/cws_train', + _cache_fp='test/demo_overwrite.pkl') + end_time = time.time() + pre_time = end_time - start_time + with open('test/demo_overwrite.pkl', 'rb') as f: + _embed, _vocab, _d = _pickle.load(f) + self.assertEqual(embed.shape, _embed.shape) + for i in range(embed.shape[0]): + self.assertListEqual(embed[i].tolist(), _embed[i].tolist()) + start_time = time.time() + embed, vocab, d = process_data_1('test/data_for_tests/word2vec_test.txt', 'test/data_for_tests/cws_train', + _cache_fp='test/demo_overwrite.pkl') + end_time = time.time() + read_time = end_time - start_time + print("Read using {:.3f}, while prepare using:{:.3f}".format(read_time, pre_time)) + self.assertGreater(pre_time - 0.5, read_time) + finally: + os.remove('test/demo_overwrite.pkl') + + def test_cache_refresh(self): + try: + start_time = time.time() + embed, vocab, d = process_data_1('test/data_for_tests/word2vec_test.txt', 'test/data_for_tests/cws_train', + _refresh=True) + end_time = time.time() + pre_time = end_time - start_time + with open('test/demo1.pkl', 'rb') as f: + _embed, _vocab, _d = _pickle.load(f) + self.assertEqual(embed.shape, _embed.shape) + for i in range(embed.shape[0]): + self.assertListEqual(embed[i].tolist(), _embed[i].tolist()) + start_time = time.time() + embed, vocab, d = process_data_1('test/data_for_tests/word2vec_test.txt', 'test/data_for_tests/cws_train', + _refresh=True) + end_time = time.time() + read_time = end_time - start_time + print("Read using {:.3f}, while prepare using:{:.3f}".format(read_time, pre_time)) + self.assertGreater(0.1, pre_time - read_time) + finally: + os.remove('test/demo1.pkl') + + def test_duplicate_keyword(self): + with self.assertRaises(RuntimeError): + @cache_results(None) + def func_verbose(a, _verbose): + pass + + func_verbose(0, 1) + with self.assertRaises(RuntimeError): + @cache_results(None) + def func_cache(a, _cache_fp): + pass + + func_cache(1, 2) + with self.assertRaises(RuntimeError): + @cache_results(None) + def func_refresh(a, _refresh): + pass + + func_refresh(1, 2) + + def test_create_cache_dir(self): + @cache_results('test/demo1/demo.pkl') + def cache(): + return 1, 2 + + try: + results = cache() + print(results) + finally: + os.remove('test/demo1/demo.pkl') + os.rmdir('test/demo1') + + +class TestSeqLenToMask(unittest.TestCase): + + def evaluate_mask_seq_len(self, seq_len, mask): + max_len = int(max(seq_len)) + for i in range(len(seq_len)): + length = seq_len[i] + mask_i = mask[i] + for j in range(max_len): + self.assertEqual(mask_i[j], j watching abc 's monday night football can now vote during for the greatest play in N years from among four or five - two weeks ago viewers of several nbc consumer segments started calling a N number for advice on various issues - and the new syndicated reality show hard copy records viewers ' opinions for possible airing on the next day 's show - interactive telephone technology has taken a new leap in and television programmers are racing to exploit the possibilities - eventually viewers may grow with the technology and the cost - but right now programmers are figuring that viewers who are busy dialing up a range of services may put down their control and stay - we 've been spending a lot of time in los angeles talking to tv production people says mike parks president of call interactive which supplied technology for both abc sports and nbc 's consumer minutes - with the competitiveness of the television market these days everyone is looking for a way to get viewers more excited - one of the leaders behind the expanded use of N numbers is call interactive a joint venture of giants american express co. and american telephone & telegraph co - formed in august the venture at&t 's newly expanded N service with N computers in american express 's omaha neb. service center - other long-distance carriers have also begun marketing enhanced N service and special consultants are up to exploit the new tool - blair entertainment a new york firm that advises tv stations and sells ads for them has just formed a subsidiary N blair to apply the technology to television - the use of N toll numbers has been expanding rapidly in recent years - for a while lines and services that children to dial and movie or music information earned the service a somewhat image but new legal restrictions are aimed at trimming excesses - the cost of a N call is set by the abc sports for example with the cheapest starting at N cents - billing is included in a caller 's regular phone bill - from the fee the local phone company and the long-distance carrier extract their costs to carry the call passing the rest of the money to the which must cover advertising and other costs - in recent months the technology has become more flexible and able to handle much more volume - before callers of N numbers would just listen and not talk or they 'd vote yes or no by calling one of two numbers - people in the phone business call this technology N - now callers are led through complex of choices to retrieve information they want and the hardware can process N calls in N seconds - up to now N numbers have mainly been used on local tv stations and cable channels - used one to give away the house that rock star jon grew up in - for several years turner broadcasting system 's cable news network has invited viewers to respond to issues should the u.s. military intervene in panama but even the hottest on only about N calls - the newest uses of the technology demonstrate the growing variety of applications - capital cities\/abc inc. cbs inc. and general electric co. 's national broadcasting co. unit are expected to announce soon a joint campaign to raise awareness about - the subject will be written into the of prime-time shows and viewers will be given a N number to call - callers will be sent educational booklets and the call 's modest cost will be an immediate method of raising money - other network applications have very different goals - abc sports was looking for ways to lift ratings for monday night football - kurt abc sports 's marketing director says that now tens of thousands of fans call its N number each week to vote for the best return etc - profit from the calls goes to charity but abc sports also uses the calls as a sales tool after callers for voting frank offers a football for $ N and N N of callers stay on the line to order it - jackets may be sold next - meanwhile nbc sports recently began scores plus a 24-hour N line providing a complex array of scores analysis and fan news - a spokesman said its purpose is to bolster the impression that nbc sports is always there for people - nbc 's consumer minutes have increased advertiser spending during the day the network 's weakest period - each matches a sponsor and a topic on unilever n.v. 's bros. sponsors tips on diet and exercise followed by a bros. commercial - viewers can call a N number for additional advice which will be tailored to their needs based on the numbers they press one if you 're pregnant etc - if the caller stays on the line and leaves a name and address for the sponsor coupons and a newsletter will be and the sponsor will be able to gather a list of desirable potential customers - an vice president says nbc has been able to charge premium rates for this ad time - she would n't say what the premium is but it 's believed to be about N N above regular rates - we were able to get advertisers to use their promotion budget for this because they get a chance to do says ms. - and we were able to attract some new advertisers because this is something new - mr. parks of call interactive says tv executives are considering the use of N numbers for talk shows game shows news and opinion surveys - experts are predicting a big influx of new shows in N when a service called automatic number information will become widely available - this service each caller 's phone number and it can be used to generate instant mailing lists - hard copy the new syndicated tabloid show from paramount pictures will use its N number for additional purposes that include research says executive producer mark b. von s. - for a piece on local heroes of world war ii we can ask people to leave the name and number of anyone they know who won a he says - that 'll save us time and get people involved - but mr. sees much bigger changes ahead - these are just baby steps toward real interactive video which i believe will be the biggest thing yet to affect television he says - although it would be costly to shoot multiple versions tv programmers could let audiences vote on different for a movie - fox broadcasting with this concept last year when viewers of married with children voted on whether al should say i love you to on 's day - someday viewers may also choose different of news coverage - a by phone could let you decide i 'm interested in just the beginning of story no. N and i want story no. N in mr. says - you 'll start to see shows where viewers program the program - integrated resources inc. the troubled financial-services company that has been trying to sell its core companies to restructure debt said talks with a potential buyer ended - integrated did n't identify the party or say why the talks failed - last week another potential buyer financial group which had agreed in august to purchase most of integrated 's core companies for $ N million ended talks with integrated - integrated said that it would continue to pursue other alternatives to sell the five core companies and that a group of senior executives plans to make a proposal to purchase three of the companies integrated resources equity corp. resources trust co. and integrated resources asset management corp - a price was n't disclosed - integrated also said it expects to report a second-quarter loss wider than the earlier estimate of about $ N million - the company did n't disclose the new estimate but said the change was related to integrated 's failure to sell its core businesses as well as other events which it did n't detail that occurred after its announcement last week that it was in talks with the unidentified prospective buyer - meanwhile a number of top sales producers from integrated resources equity will meet this afternoon in chicago to discuss their options - the unit is a constructed group of about N independent brokers and financial planners who sell insurance annuities limited partnerships mutual funds and other investments for integrated and other firms - the sales force is viewed as a critical asset in integrated 's attempt to sell its core companies - cited concerns about how long integrated would be able to hold together the sales force as one reason its talks with integrated failed - in composite trading on the new york stock exchange yesterday integrated closed at $ N a share down N cents - integrated has been struggling to avoid a bankruptcy-law filing since june when it failed to make interest payments on nearly $ N billion of debt - integrated senior and junior creditors are owed a total of about $ N billion - an earthquake struck northern california killing more than N people - the violent temblor which lasted about N seconds and registered N on the richter scale also caused the collapse of a section of the san bay bridge and shook candlestick park - the tremor was centered near southeast of san francisco and was felt as far as N miles away - numerous injuries were reported - some buildings collapsed gas and water lines and fires - the quake which also caused damage in san jose and berkeley knocked out electricity and telephones roadways and disrupted subway service in the bay area - major injuries were n't reported at candlestick park where the third game of baseball 's world series was canceled and fans from the stadium - bush vowed to veto a bill allowing federal financing for abortions in cases of rape and incest saying tax dollars should n't be used to compound a violent act with the taking of an life - his pledge in a letter to democratic sen. byrd came ahead of an expected senate vote on spending legislation containing the provision - east germany 's politburo met amid speculation that the ruling body would oust hard-line leader honecker whose rule has been challenged by mass emigration and calls for democratic freedoms - meanwhile about N refugees flew to west germany from warsaw the first in east germany 's exodus - the world psychiatric association voted at an to the soviet union - moscow which left the group in N to avoid over allegations that political were being certified as could be suspended if the of against is discovered during a review within a year - nasa postponed the of the space shuttle atlantis because of rain near the site of the launch in fla - the flight was for today - the spacecraft 's five are to the galileo space probe on an mission to jupiter - senate democratic leaders said they had enough votes to defeat a proposed constitutional amendment to ban flag burning - the amendment is aimed at a supreme court ruling that threw out the conviction of a texas on grounds that his freedom of speech was violated - federal researchers said lung-cancer mortality rates for people under N years of age have begun to decline particularly for white males - the national cancer institute also projected that overall u.s. mortality rates from lung cancer should begin to drop in several years if cigarette smoking continues to - bush met with south korean president roh who indicated that seoul plans to further ease trade rules to ensure that its economy becomes as open as the other industrialized nations by the mid-1990s - bush assured roh that the u.s. would stand by its security commitments as long as there is a threat from communist north korea - the bush administration is seeking an understanding with congress to ease restrictions on u.s. involvement in foreign coups that might result in the death of a country 's leader - a white house spokesman said that while bush would n't alter a longstanding ban on such involvement there 's a needed on its interpretation - india 's gandhi called for parliamentary elections next month - the balloting considered a test for the prime minister and the ruling congress i party comes amid charges of leadership and government corruption - gandhi 's family has ruled independent india for all but five years of its history - the soviet union from a u.n. general assembly vote to reject israel 's credentials - it was the first time in seven years that moscow has n't joined efforts led by nations to israel from the world body and was viewed as a sign of improving ties - israel was by a vote of N with N - black activist walter sisulu said the african national congress would n't reject violence as a way to pressure the south african government into concessions that might lead to negotiations over apartheid - the sisulu was among eight black political activists freed sunday from prison - london has concluded that president was n't responsible for the execution of six british in world war ii although he probably was aware of the - the report by the defense ministry also rejected allegations that britain covered up evidence of 's activities as a german army officer - an international group approved a formal ban on ivory trade despite objections from southern african governments which threatened to find alternative channels for selling elephant - the move by the convention on trade in endangered meeting in switzerland places the elephant on the list - an in colombia killed a federal judge on a street - an caller to a local radio station said cocaine traffickers had the in for the of wanted on drug charges in the u.s. - leader met with egypt 's president and the two officials pledged to respect each other 's laws security and stability - they stopped short of diplomatic ties in N - the reconciliation talks in the desert town of followed a meeting monday in the egyptian resort of - group inc. revised its exchange offer for $ N million face amount of N N senior subordinated debt due N and extended the offer to oct. N from oct. N - the n.j. company said holders would receive for each $ N face amount $ N face amount of a new issue of secured senior subordinated notes convertible into common stock at an initial rate of $ N a share and N common shares - the new notes will bear interest at N N through july N N and thereafter at N N - under the original proposal the maker of specialty coatings and a developer of technologies offered $ N of notes due N N common shares and $ N in cash for each $ N face amount - completion of the exchange offer is subject to the tender of at least N N of the debt among other things - which said it does n't plan to further extend the offer said it received $ N face amount of debt under the original offer - the stock of ual corp. continued to be amid signs that british airways may at any of the aborted $ N billion buy-out of united airlines ' parent - ual stock plummeted a further $ N to $ N on volume of more than N million shares in new york stock exchange composite trading - the plunge followed a drop of $ N monday amid indications the takeover may take weeks to be revived - the stock has fallen $ N or N N in the three trading days since announcement of the collapse of the $ 300-a-share takeover jolted the entire stock market into its plunge ever - this is a total for takeover-stock traders one investment banker said - los angeles financier marvin davis who put united in play with a $ N billion bid two months ago last night both a ray of hope and an extra element of uncertainty by saying he remains interested in acquiring ual - but he dropped his earlier $ 300-a-share bid saying he must first explore bank financing - even as citicorp and chase manhattan corp. scrambled to line up bank financing for a revised version of the labor-management bid british airways a N N partner in the buying group indicated it wants to start from - its partners are united 's pilots who were to own N N and ual management at N N - adding to injury united 's machinists ' union which helped scuttle financing for the first bid yesterday asked ual chairman stephen wolf and other ual directors to resign - a similar demand was made by a group that represents some of united 's N employees - john machinists union general vice president attacked mr. wolf as greedy and irresponsible for pursuing the buy-out - although mr. wolf and john pope ual 's chief financial officer stood to $ N million for stock and options in the buy-out ual executives planned to reinvest only $ N million in the new company - the blue-collar machinists longtime rivals of the white-collar pilots say the would load the company with debt and weaken its finances - confusion about the two banks ' efforts to round up financing for a new bid that the ual board has n't even seen yet helped send ual stock downward - and rumors of forced selling by takeover-stock traders triggered a in the dow jones industrial average around N a.m. edt yesterday - yesterday 's selling began after a japanese news agency reported that japanese banks which balked at the first bid were ready to reject a revised version at around $ N a share or $ N billion - several reports as the day gave vague or indications about whether banks would sign up - citicorp for example said only that it had of interest of a transaction from both the borrowers and the banks but did n't have an agreement - late in the day mr. wolf issued a statement calling mr. 's blast divisive and for - but he gave few details on the progress toward a new bid saying only we are working toward a revised proposal for majority employee ownership - meanwhile in another sign that a new bid is n't imminent it was learned that the ual board held a telephone meeting monday to hear an update on the situation but that a formal board meeting is n't likely to be until early next week - in london british airways chairman lord king was quoted in the times as declaring he is not prepared to take my shareholders into a deal - observers said it appeared that british air was angered at the way the bid has into confusion as well as by the banks ' effort to round up financing for what one called a deal that is n't a deal - the effort to revive the bid was complicated by the nature of the buying group - the pilots were meeting outside chicago yesterday - but british air which was to have supplied $ N million out of $ N million in equity financing apparently was n't involved in the second proposal and could well reject it even if banks obtain financing - a group of united 's employees said in a statement the fact that wolf and other officers were going to line their pockets with literally millions of dollars while severe pay cuts on the employees of united is not only but - the machinists also asked for an investigation by the securities and exchange commission into possible violations in the original bid for ual by mr. davis as well as in the response by ual - last week just before the bank commitments were due the union asked the u.s. labor department to study whether the bid violated legal standards of fairness governing employee investment funds - in his statement mr. wolf said we continue to believe our approach is sound and that it is far better for all employees than the alternative of having an outsider own the company with employees paying for it just the same - mr. wolf has merger advice from a major wall street securities firm relying instead only on a takeover lawyer peter of slate & flom - the huge drop in ual stock prompted one takeover stock trader george managing partner of & co. to deny publicly rumors that his firm was going out of business - mr. said that despite losses on ual stock his firm 's health is excellent - the stock 's decline also has left the ual board in a - although it may not be legally obligated to sell the company if the buy-out group ca n't revive its bid it may have to explore alternatives if the buyers come back with a bid much lower than the group 's original $ 300-a-share proposal - at a meeting sept. N to consider the labor-management bid the board also was informed by its investment adviser first boston corp. of interest expressed by buy-out funds including kohlberg kravis roberts & co. and little & co. as well as by robert bass morgan stanley 's buy-out fund and pan am corp - the takeover-stock traders were hoping that mr. davis or one of the other interested parties might with the situation in disarray or that the board might consider a recapitalization - meanwhile japanese bankers said they were still about accepting citicorp 's latest proposal - macmillan inc. said it plans a public offering of N million shares of its berlitz international inc. unit at $ N to $ N a share - the offering for the language school unit was announced by robert maxwell chairman and chief executive officer of london-based maxwell communication corp. which owns macmillan - after the offering is completed macmillan will own about N N of the berlitz common stock outstanding - five million shares will be offered in the u.s. and N million additional shares will be offered in international offerings outside the u.s. - goldman sachs & co. will manage the offering - macmillan said berlitz intends to pay quarterly dividends on the stock - the company said it expects to pay the first dividend of N cents a share in the N first quarter - berlitz will borrow an amount equal to its expected net proceeds from the offerings plus $ N million in connection with a credit agreement with lenders - the total borrowing will be about $ N million the company said - proceeds from the borrowings under the credit agreement will be used to pay an $ N million cash dividend to macmillan and to lend the remainder of about $ N million to maxwell communications in connection with a note - proceeds from the offering will be used to repay borrowings under the short-term parts of a credit agreement - berlitz which is based in princeton n.j. provides language instruction and translation services through more than N language centers in N countries - in the past five years more than N N of its sales have been outside the u.s. - macmillan has owned berlitz since N - in the first six months of this year berlitz posted net income of $ N million on sales of $ N million compared with net income of $ N million on sales of $ N million - right away you notice the following things about a philip glass concert - it attracts people with funny hair or with no hair in front of me a girl with sat a boy who had his - whoever constitute the local left bank come out in force dressed in black along with a of who want to be on the cutting edge - people in glass houses tend to look - and if still at the evening 's end you notice something else the audience at first and by the music releases its feelings in collective - currently in the middle of a tour as a solo mr. glass has left behind his equipment and in favor of going it alone - he sits down at the piano and plays - and plays - either one likes it or one does n't - the typical glass audience which is more likely to be composed of music students than their teachers certainly does - the work though sounds like for - philip glass is the and his music the new clothes of the - his success is easy to understand - introducing and explaining his pieces mr. glass looks and sounds more like a describing his work than a classical playing a recital - the piano which have been labeled as cyclical and are therefore therefore and but therefore both pretty and - it is music for people who want to hear something different but do n't want to work especially hard at the task - it is listening for the now generation - mr. glass has the famous less is more - his more is always less - far from being the music us with apparent not so in the of N time and or - but the music has its and mr. glass has constructed his solo program around a move from the simple to the relatively complex - opening N from the audience to the glass technique never too far from the piano 's center mr. glass works in the two on either side of middle c and his fingers seldom leave the - there is a musical style here but not a particular performance style - the music is not especially indeed it 's hard to imagine a bad performance of it - nothing no no problems challenge the performer - we hear we may think inner voices but they all seem to be saying the same thing - with planet news music meant to of allen 's wichita mr. glass gets going - his hands sit apart on the - seventh make you feel as though he may break into a very slow - the but there is little even though his fingers begin to over more of the - contrasts predictably first the music is loud then it becomes soft then you realize it becomes again - the fourth play an from on the beach is like a but it does n't seem to move much beyond its ground in three blind mice - when mr. glass decides to get really fancy he his hands and hits a bass note with his right hand - he does this in at least three of his solo pieces - you might call it a or a - in mad rush which came from a commission to write a piece of length mr. glass and confessed that this was no problem for me an a with a b section several times before the piece ends - not only is the typical it is also often multiple in its context s - mad rush began its life as the to the lama 's first public address in the u.s. when mr. glass played it on the at new york 's of st. john the - later it was performed on radio in germany and then took it for one of her dance pieces - the point is that any piece can be used as background music for virtually anything - the evening ended with mr. glass 's another multiple work - parts N N and N come from the of morris 's film the thin blue line and the two other parts from music to two separate of the story of the same name - when used as background in this way the music has an appropriate as when a phrase a minor third the seemingly endless of reports interviews and of witnesses in the morris film - served up as a solo however the music lacks the provided by a context within another medium - of mr. glass may agree with the critic richard 's sense that the N music in twelve parts is as and as the - but while making the obvious point that both develop variations from themes this comparison the intensely nature of mr. glass 's music - its supposedly a that makes one for the of the radical of and and what in even seems like in - mr. is professor of english at southern university and editor of the southwest review - honeywell inc. said it hopes to complete shortly the first of two sales of shares in its japanese joint venture for about $ N million - the company would n't disclose the buyer of the initial N N stake - proceeds of the sale expected to be completed next week would be used to repurchase as many as N million shares of honeywell stock the company said - honeywell said it is negotiating the sale of a second stake in but indicated it intends to hold at least N N of the joint venture 's stock long term - a N N stake would allow honeywell to include earnings in its results - honeywell previously said it intended to reduce its holding in the japanese concern as part of a restructuring plan which also calls for a reduction of on weapons sales - yesterday a spokeswoman said the company was pleased with our progress in that regard and hopes to provide additional details soon - honeywell said its defense and marine systems group incurred delays in shipping some undisclosed contracts during the third quarter resulting in lower operating profit for that business - overall honeywell reported earnings of $ N million or $ N a share for the three months ended oct. N compared with a loss of $ N million or N cents a share a year earlier - the previous period 's results included a $ N million pretax charge related to contract costs and a $ N million pretax gain on real estate sales - sales for the latest quarter were flat at $ N billion - for the nine months honeywell reported earnings of $ N million or $ N a share compared with earnings of $ N million or $ N a share a year earlier - sales declined slightly to $ N billion - once again your editorial page the law to conform to your almost - in an of little to his central point about private enforcement suits by environmental groups michael s. your readers the clean water act is written upon the the rather that nothing but zero risk will do it a legal standard of zero environmental sept. N - this statement surely your editorial viewpoint that environmental protection is generally silly or excessive but it is simply wrong - the clean water act contains no legal standard of zero - it requires that of into the waters of the united states be authorized by permits that reflect the limitations developed under section N - whatever may be the problems with this system it reflects zero risk or zero - perhaps mr. was confused by congress 's statement of the national goal in section N which indeed calls for the elimination of by N no less - this statement was not taken seriously when enacted in N and should not now be confused with the provisions of the statute - thus you do the public a great when mr. suggests even that the clean water act prohibits the preparation of a and water your readers may be led to believe that nothing but chance or oversight protects them as they in the night with their and waters from the knock of the sierra club at their doors - robert j. - national geographic the u.s. magazine is attracting more readers than ever and offers the glossy pages that upscale advertisers love - so why did advertising pages plunge by almost N N and ad revenue by N N in the first half - to hear advertisers tell it the magazine just has n't kept up with the times - despite renewed interest by the public in such topics as the environment and the third world it has n't been able to shake its reputation as a magazine boys like to through in search of tribe women - worse it lagged behind competitors in offering from regional editions to discounts for frequent advertisers - but now the magazine is attempting to fight back with an ambitious plan including a revamped sales strategy and a surprisingly aggressive ad campaign - advertisers do n't think of the magazine first says joan who joined in april as national advertising director - what we want to do is take a more aggressive stance - people did n't believe we were in tune with the marketplace and in many ways we were n't - the magazine has never had to woo advertisers with quite so much before - it largely on its N million subscribers in the first half up from N million a year ago an average age of N for readers at the of their years loyalty to the tune of an N N average subscription renewal rate - the magazine had its best year yet in N when it its centennial and racked up a N N gain in ad pages to N - but this year when the surrounding its centennial died so too did some advertiser interest - the reason ad executives say is that the entire magazine business has been soft and national geographic has some that make it especially during a soft market - perhaps the biggest of those factors is its high ad prices $ N for a page vs. $ N for the a comparable publication with a far smaller circulation - when ad dollars are tight the high page cost is a major for advertisers who generally want to appear regularly in a publication or not at all - even though national geographic offers far more readers than does a magazine like the page costs you an arm and a leg to develop any frequency says harry glass new york media manager for bozell inc - to combat that problem national geographic like other magazines began offering regional editions allowing advertisers to appear in only a portion of its magazines for example ads can run only in the magazines sent to subscribers in the largest N markets - but the magazine was slower than its competitors to come up with its regional editions and until last year offered fewer of them than did competitors - time magazine for example has more than N separate editions going to different regions top management and other groups - another sticking point for advertisers was national geographic 's tradition of its ads together usually at the beginning or end of the magazine rather than spreading ads out among its articles as most magazines do - and national geographic 's size means extra production costs for advertisers - but ms. says the magazine is fighting back - it now offers N regional editions it very recently began running ads adjacent to articles and it has been up its sales force - and it just launched a promotional campaign to tell chief executives marketing directors and media executives just that - the centerpiece of the promotion is its new ad campaign into which the magazine will pour about $ N mostly in the next few weeks - the campaign created by group 's ddb needham agency takes advantage of the photography that national geographic is known for - in one ad a photo of the interior of the in paris is with the headline the only book more respected than does n't accept advertising - another ad pictures a tree magnified N times with the headline for impact far beyond your size consider our regional editions - ms. says she wants the campaign to help attract advertisers in N categories including corporate financial services consumer electronics insurance and food - her goal to top N ad pages in N up from about N this year - whether she can meet that ambitious goal is still far from certain - the ad campaign is meant to the thought of national geographic she says - we want it to be a kind of image - wcrs plans sale - wcrs group hopes to announce perhaps today an agreement to sell the majority of its ad unit to eurocom a european ad executive said - wcrs has been in discussions with eurocom for several months - however when negotiations down recently wcrs 's chief executive peter scott met in paris with another french firm or - according to the executive 's involvement prompted renewed in the talks and the two agencies were hoping to out details by today - executives of the two agencies could n't be reached last night - ad notes - new account procter & gamble co. cincinnati awarded the ad accounts for its line of professional and oil products to cincinnati - billings were n't disclosed - professional products are specially made for the industry - who 's news stephen N was named executive vice president deputy creative director at grey advertising new york - he was executive vice president director of broadcast production - the commodity futures trading commission plans to restrict dual trading on commodity exchanges a move almost certain to exchange officials and traders - the cftc said it will propose the restrictions after the release of a study that shows little economic benefit resulting from dual trading and cites problems associated with the practice - dual trading gives an exchange trader the right to trade both for his own account and for customers - the issue exploded this year after a federal bureau of investigation operation led to charges of widespread trading abuses at the chicago board of trade and chicago mercantile exchange - while not specifically mentioned in the fbi charges dual trading became a focus of attempts to tighten industry regulations - critics contend that traders were putting buying or selling for their own accounts ahead of other traders ' customer orders - traders are likely to oppose such restrictions because dual trading provides a way to make money in slower markets where there is a shortage of customer orders - the exchanges contend that dual trading improves liquidity in the markets because traders can buy or sell even when they do n't have a customer order in hand - the exchanges say liquidity becomes a severe problem for traded contracts such as those with a long time remaining before expiration - the cftc may take those arguments into account by allowing exceptions to its restrictions - the agency did n't cite specific situations where dual trading might be allowed but smaller exchanges or contracts that need additional liquidity are expected to be among them - wendy the agency 's chairman told the senate agriculture committee that she expects the study to be released within two weeks and the rule changes to be completed by - the study by the cftc 's division of economic analysis shows that a trade is a trade a member of the study team said - whether a trade is done on a dual or basis the member said does n't seem to have much economic impact - currently most traders on commodity exchanges specialize in trading either for customer accounts which makes them brokers or for their own accounts as - the tests indicate that dual and traders are similar in terms of the trade executions and liquidity they provide to the market mrs. told the senate panel - members of congress have proposed restricting dual trading in bills to cftc operations - the house 's bill would prohibit dual trading in markets with daily average volume of N contracts or more those considered too difficult to track without a sophisticated computer system - the senate bill would force the cftc to suspend dual trading if an exchange ca n't show that its oversight system can detect abuses - so far one test of restricting dual trading has worked well - the chicago merc banned dual trading in its standard & poor 's 500-stock index futures pit in N - under the rules traders decide before a session begins whether they will trade for their own account or for customers - traders who stand on the pit 's top step where most customer orders are executed ca n't trade for themselves - a merc spokesman said the plan has n't made much difference in liquidity in the pit - it 's too soon to tell but people do n't seem to be unhappy with it he said - he said he would n't comment on the cftc plan until the exchange has seen the full proposal - but at a meeting last week tom the board of trade 's president told commodity lawyers dual trading is definitely worth saving - it adds something to the market - japanese firms push car - japanese luxury-car makers are trying to set strict design standards for their dealerships - but some dealers are negotiating terms while others decline to deal at all - nissan motor co. 's infiniti division likes to insist that every dealer construct and a building in a japanese style - specifications include a at the center of each showroom and a bridge a stream that flows into the building from outside - infiniti has it down to the says jay a partner at power & associates an auto research firm - toyota motor corp. 's lexus division also provides specifications - but only two-thirds of lexus dealers are new buildings according to the lexus - some are even coming up with their own novel designs - in louisville ky. for example david peterson has built a lexus dealership with the showroom on the second floor - yet some dealers have turned down infiniti or lexus because they were unwilling or unable to meet the design requirements - lee seidman of cleveland says infiniti was a bear on but at least let him an existing building without the stream - mr. seidman says he turned down a lexus franchise in part because the building was but very expensive - to head off arguments infiniti offers dealers cash bonuses and construction loans - device 's plays back a lesson - products have to be first to be winners - that 's the lesson offered through one case study featured in a design exhibit - dictaphone corp. was caught off guard in N when its main competitor office products of japan introduced a recorder half the size of standard devices - blocked by patent protection from following suit dictaphone decided to go a step further and cut the in half again down to the length of a - by N designers and engineers at dictaphone a pitney bowes subsidiary had produced a working model of a recorder - by N however the patent status of the had changed permitting dictaphone to develop its own competitive micro system which it did - marketing and sales departments then urged of the project - but others said should proceed - both were right - dictaphone went ahead and introduced the in N but it has n't sold well - to date says a dictaphone vice president it has broken even or shown a small loss - nevertheless the device has been successful in other ways - it helped dictaphone attract better engineers and it provided new technology for other company products - the recorder also helped transform the company 's reputation from to - it gave me great pride to see the inventor of the in japan look at the and shake his head and say says mr. - dictaphone 's recorder is one of N case studies in the design project sponsored by the design management institute of boston and harvard business school - the studies are on exhibit at harvard this month and will travel to chicago 's institute of design and the university of california at berkeley - a rake 's progress means out - one day carl barrett of mobile ala. was some leaves but the rake kept riding up over the - the harder he tried to push them into large the closer he came to breaking the rake and his back - so mr. barrett then vice president of the alabama association took a garden rake and taped it to the of a rake about nine inches up - his crude device worked the lower teeth gathered the leaves into a pile while the higher harder teeth moved the top of the pile - now incorporated into a rake the or also are supposed to aid in picking up leaves - one customer donald of mobile says the barrett rake allowed him to do his lawn in N N hours two hours less than usual - but other rake makers have their doubts - richard mason president of co. in w. va. says the barrett rake makes sense but it would be tough to explain to consumers - john marketing director for true corp. a subsidiary of black & decker says people do n't want to move a pile - they either pick it up he says or they start pulling from a fresh direction - odds and ends - no more or promises corp. of ind. the designer of a bed support to replace traditional - four steel each roughly in the shape of a are attached to the bottom of the box spring in a position - nearly half of u.s. consumers say they 'll pay up to N N more for packaging that can be recycled or is according to a survey commissioned by the michael peters group a design consultant - the pentagon is a house - living there for six years was really scary - the ghosts of the past are everywhere they are kept at bay only by feeding them vast quantities of our defense budget - some can be bought off relatively - during the korean war gen. douglas demanded and got in addition to his u.n. command in korea his own naval command in japan - those operations cost less than $ N billion a year and keep mac 's ghost quiet - that 's about all it costs to adm. erich 's ghost - in N and the german navy threatened to attack the panama so we created the southern command in panama - the southern command has grown even bigger since the war because 's ghost sometimes runs through the e ring dressed like gen. noriega - the command 's huge bureaucracy is needed to analyze whether leaders of coups against gen. noriega meet the war powers act 's six points cap 's seven points the intelligence committee 's N points and wilson 's N points necessary to justify u.s. support - so far no one has - the ghost of the soviet discovered in cuba back in the costs just a few hundred million the price of the caribbean command in key west that president carter created in N - the has n't been heard from since but we keep the staff around just in case - george marshall 's ghost is much more difficult to keep happy - we keep a lot of to him around the pentagon and such - the army headquarters on the third deck of the pentagon used to a lot of to him but the navy headquarters on the fourth deck made them stop it - you see marshall had this thing about the navy and the he wanted to make them part of the army but secretary of the navy james blocked him - now his ghost wo n't let up till it 's done - to keep him quiet we a new unified command every year or so run by the army or the air force and put more of the navy and under it - but we still hear him at night because the navy has a few ships left and to satisfy him the navy 's sea lift forces were given to a new air force bureaucracy in illinois its space operations to another command in colorado the to a new army bureaucracy in fort and the navy 's indian ocean and persian gulf forces to an army bureaucracy in florida - which brings up the worst and ghost of all the ghost of the shah of iran - when the shah died president carter was so scared that the shah 's ghost would blame him for him out to make way for the that he declared the carter doctrine - mr. carter said he would go to war to stop anyone from trying to grab iran - but that ghost would n't settle for words he wanted money and people lots - so mr. carter formed three new army divisions and gave them to a new bureaucracy in tampa called the rapid force - but that ghost was n't he knew the was neither rapid nor nor a force even though it cost $ N billion or $ N billion a year - after mr. carter was defeated in N the shah 's ghost claimed the credit and then went after president reagan and cap - i saw what he did to them - it made my dance with - why he used to lay in wait for cap suddenly he 'd leap from behind some of marshall onto cap 's and grab him by the and him till he up an additional $ N billion or so - cap added four more divisions to the army two active and two reserve two carrier groups to the navy a division equivalent to the and the and a thousand tactical aircraft to the air force - he bought $ N billion in ships and $ N billion in and equipment to fill them and them at a new $ N billion base at diego garcia in the middle of the indian ocean - he dedicated all these new forces to the persian gulf - one night both marshall 's ghost and the shah 's ghost together caught cap and threw him to the ground - before they let him go he added a thousand bureaucrats to the in tampa and renamed it central command - he gave those bureaucrats charge of all naval operations in the persian gulf and indian ocean - marshall figured it would be good training for those soldiers someday maybe they would get the whole navy - they had fun moving the carriers around but it turned out that they had forgotten all about mine - but the shah still kept leaping out at cap so cap bought a hundred merchant ships more and $ N billion of etc. in order that those seven new army divisions and three marine could unload from all those new ships and aircraft and go to war in the - then suddenly 's ghost came to visit and said what the hell are you doing planning for a land war in asia N miles away - we 'd get our kicked - lucky for cap was and soon went away while the shah he kept coming back - so the u.s. found itself paying about $ N billion in to various arab for rights around the indian ocean - we had great success in somalia - but then it turned out that president was not at all a nice person and the navy pointed out that the base he promised us in had up about a hundred years ago and anyway was N miles from the mouth of the gulf - but who 's counting - still was the best we could get so we stay in bed with president - all these reports about him committing are probably anyway - but would n't you know now that we are spending of dollars and have built those new divisions and new air wings and have positioned all these ships and supplies to fight the russians in iran the russians seem to have lost interest in the whole subject - meanwhile congress is cutting huge chunks out of the rest of the defense budget - predictably some navy guys said do we still need to keep all N army divisions on active duty and all those extra aircraft without bases and all those army guys playing in tampa - could n't we save $ N billion or $ N billion a year by shifting that stuff to the reserves - and why not save the costs of a thousand bureaucrats by central command and putting responsibility for gulf naval operations back where it belongs afloat with the task force in the gulf - and where were all our paid indian ocean allies last year when our were being attacked - questions like that really stir up marshall 's ghost - he appeared late one night in the of the new defense secretary dick cheney - marshall came in like 's ghost dragging those chains of and air wings and links with arab - he would n't leave until mr. cheney promised to do whatever the pentagon systems analysts told him - so next day mr. cheney went out and did just that he canceled the navy and cut back one carrier and N - then he canceled production of the navy 's most important carrier aircraft the f-14 and the - on the other hand mr. cheney retained all those new land forces - marshall 's ghost is satisfied for now but he 'll be back - what with halloween coming and bigger defense cuts looming more and more pentagon bureaucrats are under their desks - they know that they can hold off the ghosts only a little while longer by cutting carriers and ships - then the whole thing will start to collapse just as it did in the 1970s and the ghosts and will be through the place turning people 's hair white - gives me the just thinking about it - mr. lehman a reagan navy secretary is a managing director of painewebber - the metal and marble lobby of centrust bank 's headquarters is than your average savings and loan - for one thing there is an old master on the wall samuel david a big painted by a - at the moment however the painting is a nagging reminder of the problems that have centrust and its flamboyant chairman and chief executive david l. paul - in an international buying spree that began barely two years ago mr. paul a collection of about N works including the at a total cost of $ N million - by midnight oct. N all of the paintings were supposed to have been sold off under orders from florida 's comptroller whose office the state 's s&ls - centrust did n't meet the deadline - the collection was at the heart of a plan mr. paul had in which the art was to do double duty as an investment for centrust and as for the s&l 's new office tower designed by - the is that the $ N million was from the funds of this federally insured institution even as centrust was losing money hand over - mr. paul had no right to buy art for the s&l in the first place it is n't on the comptroller 's permissible list without seeking a special which he did not do - besides that some of the paintings that were to grace the walls of centrust actually ended up hanging in the chairman 's estate on la off miami beach - last spring the comptroller 's office called a halt to mr. paul 's giving him six months to sell the paintings - the acquisitions officials said in a letter to mr. paul were and unauthorized - so far mr. paul has but three of his he wo n't say to whom - the comptroller 's office says it is monitoring the situation - though the agency could remove mr. paul it has no current intention to do that - it 's not like selling mr. paul says as he takes a drag on a st. cigarette - the last six months has established the quality of the collection - there 's no fire sale here - despite mr. paul 's characteristic the is finding that getting centrust florida 's largest thrift institution out of its investments is much tougher than getting into them had been - paintings are just part of the picture - although mr. paul has a $ N billion junk-bond portfolio to less than $ N million since april the high-yield debt market has plummeted - itself of what is left as is required of all thrift institutions by july N under the new federal s&l bailout law may well prove difficult - and centrust has other problems - late last week federal regulators ordered the thrift institution to stop paying dividends on its preferred stock a move that suggests deep concern about an institution - mr. paul has a plan to bring in $ N million by selling off N of centrust 's N branches but it has yet to be approved by regulators - it is mr. paul 's art venture however that has drawn the most attention from investors and regulators not to mention throughout the world - shareholders some of whom are suing say the chairman and his collection the excesses of speculation that set off the national s&l crisis - centrust shares have fallen sharply in price from a high of $ N in N to close yesterday at $ N - gallery directors meanwhile say mr. paul and others of his have left an mark on the art world and not for the better - collectors do n't say it 's a van anymore harry brooks the president of & co. a new york gallery - they say got $ N million for his so certainly $ N million is n't too much for mine - the great collectors we depended on such as paul mellon or norton simon have stopped buying and the new buyers are brilliant men who made money in the stock market or in takeovers and rushed into collecting - mr. an art dealer and sold vincent van 's at a sotheby 's auction in november N to australian businessman alan bond - trouble is mr. bond has yet to pay up and until he does sotheby 's has the painting under lock and key - when mr. paul moved in on the art market he let it be known that virtually no piece was too costly to be considered by centrust - he established his reputation as a in january last year at sotheby 's auction of the linda and gerald guterman collection in new york - there on one of his first shopping trips mr. paul picked up several paintings at stunning prices - he paid $ N million for instance for a still life by jan that was expected to fetch perhaps $ N - the price paid was a record for the artist - some N N of items offered at the guterman auction were sold at an average price of $ N - the rest were withdrawn for lack of acceptable bids - afterward mr. paul is said by mr. guterman to have mr. guterman the new york developer selling the collection and - he says he them recalls mr. guterman - and he tells me if you want to see your paintings you 'll have to come to my house in florida - mr. paul denies and - it 's just not true he says - mr. paul quickly became more aggressive in his collecting with the help of george wachter a sotheby 's expert in old masters whom he met at an exhibition of the guterman items - mr. wachter who became his principal adviser searched in london paris and - and according to one dealer mr. wachter had a for introducing mr. paul with the phrase he can buy anything - nicholas hall the president of the u.s.a. ltd. gallery in new york sold mr. paul and in the by giovanni - mr. hall says mr. paul was known to spend a lot of money - people were interested in seeing him but it was recognized that the route was through sotheby 's and particularly george wachter - mr. paul thus developed a close relationship with sotheby 's - mr. paul was eager to a collection for the headquarters centrust has been moving into for the greater part of a year - sotheby 's the auction house founded in london N and now under the of sotheby 's holdings inc. was hoping to stir up interest in old masters as it to build its u.s. business - european dealers continued to dominate the action in old masters which sotheby 's north america had lately been touting in this country - for several months there was optimism all around - last october mr. paul paid out $ N million of centrust 's cash plus a $ N million commission for portrait of a man as - the painting attributed to artist peter paul rubens was purchased privately through sotheby 's not at auction - in march N just N months into his campaign mr. paul was named by art & magazine as one of the top N individual collectors in the u.s. - an unknown quantity to most of the art world paul is no to spending the magazine said noting that he does n't stop at on but also spends big on art you can eat - he recently bid $ N at a paris charity auction for a dinner by six of the world 's great chefs but the final party cost closer to $ N - mr. paul says it was n't that high - the art collection might have come to rival the ' had the florida comptroller 's office not got wind of mr. paul 's - in its letter to him dated march N and shared with reporters alex the chief of the bureau in the comptroller 's office expressed that the s&l could be so when it had reported losses of more than $ N million in its two preceding quarters - the state gave centrust N days to sell the rubens - the comptroller 's office eventually extended the deadline to six months but its demands ordering that the book value of the collection be reduced to zero - in other words get rid of all the pictures - the state noted that banking practices are grounds for removing an officer or director and closed with the to mr. paul govern yourself - the state agency was particularly to learn that the rubens and a other paintings listed among the bank 's furniture and were actually hanging in the chairman 's house - mr. paul says that at one point he did indeed have eight or nine of the paintings at home and that the rest were in storage at sotheby 's - he explains that he was merely the paintings at home with some display because of the special environment required for their until centrust 's new building was ready for them - still the incident was embarrassing - it came on the heels of a number of local newspaper articles suggesting that mr. paul has benefited from his association with centrust - for instance he got a $ N million loan from the s&l negotiated at a rate - he owns N N of centrust 's shares - adding to mr. paul 's problems dealers some with vested interests insist that he relying rather too heavily on sotheby 's advice paid much too much for several pieces in the centrust collection - the $ N million on the rubens for example was a record price for the artist and maybe twice its value given a dispute among scholars about its - david the president of david inc. a new york gallery says scholars question the of the rubens - it may have been painted instead by a rubens associate - the feeling among many experts on the commercial side is that the price paid at the time was excessive in any event mr. says - it sounds like with the rubens he got absolutely taken to the - victor the executive director of the association of america agrees that mr. paul paid very for the rubens and adds that getting rid of it any time soon for a similar sum would be quite a feat - it 's not beyond credibility the rubens will someday be worth $ N million but whether it could be sold for that amount tomorrow remains to be seen - still predicting is tricky - i 'm forever by what i see making these high prices - jonathan h. the son of the painting 's former owner mrs. rush the price talk as sour - dealers of the purchase price he says were themselves interested in buying the rubens but lost out - mr. paul for his part the rubens price saying a lot of the experts have never seen the thing itself - most of them were n't even born the last time the painting was displayed publicly he says - art prices are but a good deal of is involved in statistics on sales - salomon brothers inc. the investment-banking firm in its annual tally of investment returns reported that old masters N N in the year ended june N the greatest return of any of N assets it tracked - and modern paintings not tracked by salomon are ranked even higher at N N by sotheby 's - salomon moreover gets its data on art appreciation from sotheby 's whose prices go up with clients like mr. paul in its - the from consideration the many paintings that go at auction - art indexes track winners not losers - but art that has fallen sharply in value is rarely put up for sale - also at any of sotheby 's auctions of old masters roughly one-third to of what is offered does n't sell at any price - it 's not that there are n't any bids but the bids do n't meet the minimum reserve prices set by the sellers - in january the painting that now hangs at centrust was expected to bring no more than $ N at auction until mr. paul came along with his $ N million - mr. hall of the gallery says $ N million would have been an impossible price for anyone to ask for a four years ago - but from his point it is n't that mr. paul a customer of his too for the work a painting by an artist who is not a household word - the painting is N feet wide seven feet high - rather it just shows things have changed - mr. paul boasts that he spotted bargains in old masters just before they took an upward turn - they went up N N last year and they 'll do it again this year he declares - they were a - everybody was out buying - sotheby 's vice president says the auction house has been mr. paul in selling the paintings - and while sotheby 's chief rivals in the art world private art dealers wo n't be happy to hear it she adds a number of the have already been sold and at a substantial profit - mr. paul claims to have sold three paintings at more than a N N profit - that is n't N N and the claim is n't - he furthermore denies that he relied too heavily on sotheby 's or mr. wachter - mr. paul says he had not one but four advisers and that he never bid - after all he had the counsel of from the most reputable in the world - he says he expects to sell the collection including the controversial rubens carefully and just as it was put together - but in mr. paul 's holdings are - that is he is being to put them on the market too soon and has already gotten offers that are less than he paid for some of the art works - after a few years you can argue there has been natural appreciation says susan the publisher of leonard 's annual price index of art auctions - but quick turnover in is like your jewelry you end up with N N - people hold out and try to get a bargain - sotheby 's itself and mr. paul in the matter - mr. wachter says mr. paul was a quick study who worked intensely and bought the best pictures available at the moment - on occasion he paid a high price mr. wachter concedes but he says those who bid less and dropped out were dealers who would then have marked up the paintings to them at a profit to collectors - a at associates in san francisco considers it conflict of interest for an auction house to both advise a client on purchases and to set price estimates on the paintings to be purchased - sotheby 's she says is wearing both hats - i ca n't see why there would be a conflict of interest says sotheby 's ms. - estimates are based on the previous price of similar works sold at auction and current market conditions and are not affected by any knowledge of who the potential buyer could be - frequently clients express interest in paintings but do n't end up bidding she adds so we do n't know who the potential buyer will be - mr. paul in selling off his paintings is seeking at least a N N return on the bank 's investment so as to prove that the venture was sound - mr. paul says that he has out over much of the globe and that potential buyers from as far away as japan and italy have examined the collection - because of the pressure on centrust to sell dealers and collectors have been trying to get the paintings at prices - but so far mr. paul and his advisers are holding fast - one dealer martin of french & co. in new york says he would have loved to buy a jan de painting from the bank - i tried to steal the picture to buy it and sotheby 's would n't do it - they were protecting his interests - meanwhile mr. paul and centrust executives are getting about - mr. paul has been characterized as the great or something complains karen e. an executive vice president of centrust - the media she says have distorted his personal life - mr. paul in agreement - i do n't think i have a life style that is frankly so flamboyant he says - but at just that moment he is interrupted in his office by a in who coffee from silver into a cup of china and the with - mr. paul says yes the ceiling in his executive is - the offices are done in and books and of course a $ N million rubens - but he that the be played down - do n't say it 's a gold ceiling - just say the offices are appointed he says - otherwise the regulators will take it for and everything 's got to be - figures do n't include taxes or transaction costs - companies listed below reported quarterly profit substantially different from the average of analysts ' estimates - the companies are followed by at least three analysts and had a minimum five-cent change in actual earnings per share - estimated and actual results involving losses are omitted - the percent difference compares actual profit with the 30-day estimate where at least three analysts have issues forecasts in the past N days - otherwise actual profit is compared with the 300-day estimate - during its centennial year the wall street journal will report events of the past century that stand as milestones of american business history - creative accounting mostly by forced to change their way of setting standards to be followed by corporations reporting financial results standards that had become all too flexible - the new financial accounting standards board fasb was created in N to replace the accounting principles board of the american institute of certified public accountants - all of the former board 's members were criticism because they were writing rules while handling clients ' books at the same time - the new board 's structure kept four but the others were from industry and - francis m. wheat a former securities and exchange commission member headed the panel that had studied the issues for a year and proposed the fasb on march N N - the former board had produced N opinions and N critics in its 12-year life its chairman had conceded - the climate was right for the new fasb - in the late 1960s some failed to correct such abuses as clients picking rules that earnings and stock prices - and in november N congress had passed a special act to one board rule - also james needham an sec commissioner in april N had warned that the industry might face a federal agency writing accounting rules if they rejected the fasb idea - of the books dubbed figure the threat - the fasb had its initial meeting on march N N - on dec. N N it issued its first rule it required companies to disclose foreign currency in u.s. dollars - the fasb since then has issued N rules and some still industry - since late N for example it has put off a rule dealing with deferred income taxes because of the continuing controversy over the issue - industrial corp. said it plans to repurchase N shares or about N N of its shares outstanding in open market transactions - the metal products concern currently has N million common shares outstanding - previously had said it planned to repurchase shares but did n't disclose when or how many shares it intended to buy back - the company named dillon read & co. as its exclusive agent for the stock buy-back program - a seat on the chicago board of trade was sold for $ N down $ N from the previous sale last tuesday - seats currently are quoted at $ N bid $ N asked - the record price for a full membership on the exchange is $ N set aug. N N - an associate member seat was sold for $ N up $ N from the previous sale oct. N - associate member seats currently are quoted at $ N bid $ N asked - the record price for associate membership is $ N set aug. N N - industries ltd. said its link flight division was awarded a contract by the u.s. army for two helicopter which the company valued at as much as N million canadian dollars us$ N million - said the fixed price for the first of the combat mission is c$ N million - it is scheduled for delivery in late N - the price of the second ranges between c$ N million and c$ N million said depending on when the army exercises its option - is a toronto-based maker of commercial and military aircraft and training equipment - inc. said it agreed to team with a unit of honeywell inc. to provide power for a new military system being proposed by honeywell - total value of the contract could be $ N million said and work on the project would be about evenly divided - as previously reported emerged from chapter N bankruptcy-law protection in february - this los angeles company and its union federal savings bank subsidiary said more than N N of their N N N convertible subordinated debentures due N were tendered for conversion into common stock - the conversion increased total equity capital by about $ N million to a total of $ N million - union federal a federally insured savings bank has $ N billion in assets - david d. lung was appointed president and chief operating officer of this maker of building materials for manufactured homes and recreational vehicles - as president mr. lung N years old succeeds his father d. lung N who founded the company in N - lung remains chairman and chief executive officer - david lung has been with patrick since N and has served as vice president for administration and purchasing since N - general dynamics services co. a unit of general dynamics corp. won a $ N million army contract to establish maintenance facilities for tracked vehicles in pakistan - grumman corp. was given a $ N million navy contract for improvements - hughes aircraft co. a unit of general motors corp. got a $ N million air force contract for equipment - reynolds metals co. said third-quarter net income dropped nearly N N to $ N million or $ N a share from $ N million or $ N a share a year earlier - the latest earnings reflect an increase of about N million in common shares outstanding - revenue rose N N to $ N billion from $ N billion - reynolds is the third big aluminum company since friday to report disappointing earnings - the no. N domestic aluminum producer aluminum co. of america friday said its earnings fell N N to $ N million or $ N a share - and ltd. yesterday reported net income slid N N to $ N million or N cents a share from $ N million or $ N a share - analysts on average had been expecting about $ N for and $ N for - it 's a good indication that level of profitability has peaked for the industry says metals analyst with ball & inc. who had estimated reynolds would earn about $ N a share - the nation 's no. N aluminum company said earnings were hurt by lower prices for certain aluminum products which typically follow price fluctuations of primary - the base metal price has dropped N N from a year earlier to N cents a pound - much of the price decline has been blamed on a slowing economy and the third quarter is typically the industry 's period - but william o. chairman and chief executive officer said the price appears to have out - he said shipments are continuing at a healthy pace and the company has no excess inventory - aluminum shipments of N metric tons were nearly equal to the year-earlier period the company said - nevertheless the company said that in the latest quarter there were increased material and labor costs including a new employee plan - in composite trading on the new york stock exchange reynolds closed at $ N up $ N - no but certainly no home run - that 's how the game is shaping up for the months ahead according to money managers and a few brokers - yesterday 's recovery from friday 's in the dow jones industrials had many brokerage houses that stocks are a good bargain again - but quite a few money managers are n't buying it - weakening corporate earnings they say are no prescription for a bull market - the stock market ai n't going to do much of anything for a while says john of wellington management who runs the $ N billion windsor fund - he suspects that friday 's market decline may have a second leg perhaps a N N to N N drop later on - mr. says the stock market has lost some powerful driving forces namely earnings growth and the lbo buy-out fever that investors to bid up whole groups of stocks such as media and airlines - after sitting with N N of his fund in cash before friday 's sell-off mr. says he bought a narrow list of stocks yesterday - with flat corporate profits on the horizon for N money managers say price-earnings multiples that look cheap today might go on being cheap for a long time - this is not a grossly market but it 's not cheap either says george collins president of the mutual fund company t. rowe price associates in baltimore - according to institutional brokers estimate system wall street market strategists see only a N N jump in company profits in N unlike in N when profits a year out looked good they did soar N N in N - bulls say the market is an incredible bargain priced at only about N times estimated N earnings for stocks in the standard & poor 's N index - before the N crash the was more than N - the common view says cohen strategist for drexel burnham lambert is that there will be mild economic growth modest profit expansion and things are going to be - our view is that we may see a profit decline - some think investors should sell into rallies - the market is going to wind down says gerald w. a chicago money manager - things are a little less after friday 's jolt in the market - he expects stocks to decline an additional N N to N N with the dow perhaps out between N and N between now and june - after friday 's decline mr. 's firm ran statistical tests on N high-quality stocks using old-fashioned value criteria devised by benjamin graham an analyst and author in the 1930s and who is widely considered to be the father of modern securities analysis - he found N still and N fairly valued - nicholas parks a new york money manager expects the market to decline about N N - i 've been two-thirds in cash since july and i continue to think that having a defensive position is appropriate he says - companies that on debt in leveraged buy-outs during the past two years will continue to surface as business problems - about value are n't useful says new york money manager john of delta capital management - for instance he says international business machines and unisys might look cheap but investors might continue to do better with stocks like walt disney procter & gamble and coca-cola strong performers in recent years - money manager robert ross head of ross associates ltd. in vancouver british columbia says stocks would have to fall N N to N N before they are competitive with less risky investment alternatives - russell a money manager in okla. says friday 's is going to have more of a permanent impact on the of many investors than wall street would want to admit - there are still bulls out there - i still think we will have a N dow whether it 's six months or N months from now i do n't know says david managing partner of value management in new york - we 're doing a little buying in some stocks that have really been down - many brokerage house officials also are optimistic - yesterday goldman sachs merrill lynch and dean witter all increased the proportion of assets they recommend investors commit to stocks - dean witter now recommends N N goldman N N and merrill lynch N N - some investors say friday 's sell-off was a good thing because it a lot of crazy takeover speculation - it was a healthy says michael who runs salomon brothers asset management in new york - from here out these investors see a return to old-fashioned investing based on a company 's ability to show profit growth - the fundamentals are pretty strong mr. says - i do n't see this as a bear market at all - it 's a recognition that there was much too much in the lbo market - friday 's big fall was just a by the stock market says john connolly chief strategist for dean witter - it was an to an event the failure of a management and union group to get bank financing for a takeover of ual that does n't mean that much to lots of stocks - many investors have nagging worries however - newspapers are full of about companies on their debts and banks writing off real estate loans - that investors ' confidence in the economy and stocks - not even all the brokerage firms see clear sailing ahead - disappointing profits are likely to get worse in the next two quarters says mary farrell a market strategist at painewebber - she thinks the market could drop about N N in the next few months then recover and go higher - companies with steady earnings growth could do well she says while others with high debt or poor earnings could see their shares decline far more than N N - the turmoil on wall street may benefit some retailers attempting to lead leveraged buy-outs of their specialty and department-store chains investment bankers and retailers said - managers at five chains have said in recent weeks that they intend to bid for their companies - the chains include bloomingdale 's owned by campeau corp. toronto saks fifth avenue and marshall field 's owned by b.a.t industries plc london and b. altman & co. and inc. owned by hooker corp. which is now being managed by a court-appointed provisional - hooker is based in sydney australia - the combination of so many chains available for sale the recent failures of such retailing lbo 's as miller & inc. and declining investor confidence will drive down prices retailing observers said - the pricing will become more realistic which should help management said bruce rosenthal a new york investment banker with nathan s. & co - investors are n't going to be throwing money at any of the proposed lbos but doing deals on the basis of ridiculous assumptions never made sense either - earlier this year bankers and other investors were willing to provide financing because they assumed there would be major gains in both profitability and sales mr. rosenthal added - those days are over now he believes - competition from third parties who have cash and are prepared to buy has always existed and will continue added mr. rosenthal - but when prices were crazy it was even harder to do an lbo - bankers believed in the theory that says somebody else is always willing to pay more - this is no longer true today - at saks fifth avenue paul senior vice president marketing agreed that lower prices will help his management team in their proposed lbo - having to take on less debt would certainly be an advantage said mr. - it would also help us in our search for equity partners - to make an lbo work now we are going to need more than just junk bonds - none believe the proposed management lbos will be easy to complete especially at b. altman & co. which is under chapter N bankruptcy protection - not only could the wall street gyrations damp christmas sales if consumers lose confidence in the economy but potential junk-bond buyers are sure to demand even stronger and greater management equity participation - further many institutions today holding troubled retailers ' debt securities will be to consider additional retailing investments - it 's called bad money driving out good money said one retailing - institutions that usually buy retail paper have to be more concerned - however the lower prices these retail chains are now expected to bring should make it easier for managers to raise the necessary capital and pay back the resulting debt - in addition the fall selling season has generally been a good one especially for those retailers dependent on apparel sales for the majority of their revenues - what 's encouraging about this is that retail chains will be sold on the basis of their sales and earnings not liquidation values said joseph e. brooks chairman and chief executive officer of ann taylor inc. a specialty chain - retailers who had good track records of producing profits will have a better chance to buy back their companies - still most retailing observers expect that all the proposed retailing lbos will depend partly on the sale of junk bonds a market already in in part because of concerns associated with bonds issued by the federated and allied units of campeau - prices for retail chains are lower today than they were last week which will help management said harrison chairman of inc. an investment-banking firm specializing in retailing acquisitions - but the hurdle of financing still has to be resolved - potential bondholders will either look for greater equity participation on behalf of management or insist the equity component of the deals be substantially greater than in the past - sony corp. won a pretrial order blocking u.s. sales of justin products inc. 's my own line of portable audio players for children - judge john e. issued the order in manhattan federal court where sony has accused the tiny company of illegally knocking off the my first sony line - the judge held that the combination of colors used for the sony products is distinctive and subject to protection under new york state law rather than federal law - the legal fight was the subject of a wall street journal story yesterday - justin 's attorney charles e. said justin would ask an appeals court to set aside the order temporarily pending an appeal - he also repeated justin 's of sony 's charges - their likelihood of us is very slim said lewis h. sony 's attorney who said he doubts justin will go ahead with a trial - continental mortgage & equity trust said it will resume dividend payments with a payout on nov. N to shares of record oct. N - the dallas real estate investment trust last paid a dividend on dec. N N when shareholders received $ N a share - despite continuing troubles with problem assets and nonperforming loans the trust said it expects to be able to maintain or increase the rate of distributions because of operations of joint-venture properties - a federal appeals court struck down a natural-gas regulation that had prevented pipeline companies from passing to customers part of $ N billion in costs from controversial contracts - the court in a N ruling threw out a deadline set by the federal energy regulatory commission for settling old contract disputes over gas that the pipeline companies reserved but did n't use - ferc 's regulation had given pipelines until march N N to pass on to customers as much as N N of the costs of buying out their broken contracts which were made with producers when gas prices were high and supplies short - a majority of old contracts were by the deadline and settled at steep discounts - but pipeline companies estimate they still face $ N billion in liabilities from disputes including $ N billion they fear they wo n't be able to pass on to customers - according to industry lawyers the ruling gives pipeline companies an important second chance to resolve remaining disputes and take advantage of the cost-sharing mechanism - the court left open whether ferc could a new deadline later - the court agreeing with pipeline companies found the march N deadline was and and highly to the bargaining power of pipelines that were forced to negotiate settlement of the old contracts to meet the deadline - a report last month by the interstate natural gas association of america found that pipelines ' settlement costs had jumped in the three months before the deadline to N cents on the dollar from N cents on the dollar in N - the court ordered ferc to justify within N days not only its cost-sharing deadline but other major elements of its proposed regulation for introducing more competition into natural-gas transportation - the court also questioned a mechanism that could be used to resolve liabilities - the complex regulation known in the industry as order N has been contested by all sides including natural-gas producers pipelines local distribution companies and consumers - the court 's decision would allow ferc to change some of its provisions but it will be reviewed again quickly by the court - corp. said it voluntarily prepaid $ N million on its original $ N million term loan bringing the total debt reduction for the year to $ N million - after the payment the cleveland company owes $ N million on the loan - the cement producer said the payment was made from excess cash flow - national income realty trust said it will resume dividend payments with a dividend to be paid nov. N to shares of record oct. N - the mortgage and equity real estate investment trust last paid a dividend on aug. N N when holders received N cents a share - despite continuing troubles with problem properties and nonperforming loans the dallas trust said it has reserves abandoned properties with little potential and experienced improved operating results from joint ventures - mlx corp. said it reached a preliminary agreement with senior lenders to its and group to restructure the $ N million of credit facilities the lenders provide to the group - mlx which also makes aircraft and truck parts said the debt was accumulated during its acquisition of nine businesses that make up the group the biggest portion of which was related to the N purchase of a co. unit - among other things the restructured facilities will substantially reduce the group 's required amortization of the term loan portion of the credit facilities through september N mlx said - certain details of the restructured facilities remain to be negotiated - the agreement is subject to completion of a definitive amendment and appropriate approvals - william p. mlx chairman and chief executive said the pact will provide mlx with the additional time and flexibility necessary to complete the restructuring of the company 's capital structure - mlx has filed a registration statement with the securities and exchange commission covering a proposed offering of $ N million in long-term senior subordinated notes and warrants - dow jones & co. said it acquired a N N interest in corp. a subsidiary of oklahoma publishing co. oklahoma city that provides electronic research services - terms were n't disclosed - customers of either or dow jones are able to access the information on both services - dow jones is the publisher of the wall street journal - flowers industries inc. said it will report a charge of eight cents to N cents a share for its fiscal first quarter ended sept. N from the sale of two in high point n.c. and - the company said it sold the to mills family for an undisclosed amount - it said the sales were part of a N federal trade commission consent order - a year earlier flowers had fiscal first-quarter net income of $ N million or N cents a share on revenue of $ N million - production by the nation 's mills decreased N N last week to N tons from N tons the previous week the american iron and steel institute said - last week 's output rose N N from the N tons produced a year earlier - the industry used N N of its capability last week compared with N N the previous week and N N a year ago - the capability utilization rate is a designed to indicate at what percent of its production capability the industry is operating in a given week - b. was named executive director of the commission effective early november - mr. N years old succeeds N who resigned to join hong kong 's securities and futures commission - mr. was vice president and director corporate finance of thomson inc. a toronto investment dealer - dun & bradstreet corp. 's market data unit said it acquired school and college construction reports service from intelligence for education inc - terms were n't disclosed - the service supplies weekly reports on school and college construction plans - market data is a of educational information and provides related services - closely held intelligence in education of n.y. is an educational publisher and consultant - a battle is in venice over plans to have the italian city be the site for a universal in N - the plans include a subway system a congress center floating trees and as many as N additional tourists a day - enthusiasts argue that holding the fair would attract businesses create jobs and help abandoned sections of town - but opponents fear - this city already has too many tourists and it ca n't hold them all says the president of the venice association - about N italian businesses including fiat s.p a. and c. olivetti & co. have formed a consortium to lobby for holding the in venice - three gambling casinos have opened in poland - the three two in warsaw and one in accept only foreign currency and are joint ventures between polish firms and western companies - not all poles are pleased - what do we want casinos for when we have n't got anything in the shops one asked - but who runs the casino at warsaw 's hotel said the ventures would help poland service its $ N billion foreign debt by pouring dollars into the state firms in the joint ventures the lot airline and tourist organization - plans to increase natural-gas sales to europe and the u.s. - according to the middle east economic survey the north african nation is holding talks with italy for adding a fourth pipe to a section of the pipeline expanding capacity by up to six billion cubic a year from N billion - also wants to build a pipeline through and across the of to supply spain france and west germany with up to N billion cubic a year by the late 1990s - south africa 's national union of agreed to suspend the strike by diamond workers and resume negotiations with de beers consolidated mines ltd. over their wage dispute de beers said - it also said the union had agreed to meet the company for further talks tomorrow - the strike at five de beers mines began last thursday with N out of a total N members employed on de beers mines participating according to the union while de beers said there were N participants - the union has demanded a N N increase in the minimum wage while de beers 's final offer was an increase of N N - a environmental conference opened in - the gathering is expected to focus on curbing the of and limiting damage from industrial and improving the handling of harmful chemicals - west german environment minister said bonn is convinced of the need for cooperation especially with our neighbors in the east because we are directly affected by their ecological progress or lack of it - the u.s. and canada joined every european country except at the meeting - the swedish publishers of a new newspaper rushed an extra edition across the on oct. N after the first run sold out in one day - editor said plans had called for N copies of the monthly are business paper to be sold at and an additional N promotion issues to be sent by direct mail - he said N more copies were sent to because of strong sales - the swedish publishing company owns N N of are and the management company minor owns N N - mexico 's top debt negotiator said the country 's creditor banks are responding to mexico 's package - mr. 's optimism contrasts with some bankers ' views that the deal may require a lot of arm by the u.s. treasury in order to succeed - mr. mexico 's of the ministry of finance met yesterday with european bankers in london at the point on a so-called road show to market the package around the world - an increasing number of banks appear to be considering the option under the deal they can swap their mexican loans for 30-year bonds with a face value discounted by N N mr. said - the other two options consist of loans for bonds with N N interest rates or providing fresh loans - the accord which covers $ N billion of mexico 's medium and long-term debt is expected to go into effect in early - china 's top film actress paid $ N in back taxes and fines in province the people 's daily reported - the amount is equal to about N years earnings for the average peasant who makes $ N a year - china will spend $ N million for maintenance on 's palace former home of the lama the china news service said - the lama who was just awarded the nobel peace prize lives in in india - george w. koch N years old president and chief executive officer of grocery manufacturers of america inc. was elected a director of this maker of and specialty foods succeeding n. white jr. N who resigned - american business computer corp. said it privately placed N common shares at $ N a share - the placement was made through gray securities new york to institutional investors - proceeds will be used to recently technology and support the company 's international expansion - the company develops and markets products for the food service industry - the r.h. macy & co department-store chain is n't for sale - in yesterday 's edition it was incorrectly included with a list of new york chains up for sale - korean car exports have slid about N N so far this year but auto makers here are n't - they are enjoying domestic sales that are more than making up for lost overseas sales - south korean consumers are expected to buy almost N passenger cars this year up N N from N - in fact some auto executives suggest that demand for their cars in the u.s. and canada is a blessing otherwise they would n't be able to keep up with demand in the more profitable local market - we are very lucky to easily change an export loss to domestic plus says hong managing director of domestic marketing for hyundai motor co - as it is waiting lists of a month are n't unusual for popular models - demand is so strong that all of the domestic makers hyundai kia motors corp. daewoo motor co. and even ssangyong motor co. plan to build more factories - industry analysts predict that by N south korea will be building three million cars a year about half of that for export - it 's an optimistic move in a industry already facing world-wide overcapacity - but south korean auto makers are confident that the export market will bounce back and that demand in korea will stay strong - currently only one in N south koreans owns a car up from one in N a decade ago - in the year N it will be one car per family - at that point domestic sales will slow down says kim director of marketing for daewoo motor - the reason for the tremendous demand is simple south koreans suddenly have a lot more money - we never thought we 'd own a car says ok who just bought a daewoo on a five-year loan - she and her husband started a small printing business and need the car for work as well as for weekend - pay raises of N N over the past three years have given many south koreans the money to enjoy the things they were supplying the rest of the world - the success of ssangyong motor shows the strength of the auto market and its growing diversity - a part of the conglomerate ssangyong group it took over the dying motor co. in N - ssangyong began making variations of the vehicle - had had a technology agreement with jeep maker american motors corp. now a part of chrysler corp - the most popular style is the stretched family which resembles a ford or chevy - the vehicles start at $ N a family can cost over $ N - ssangyong which has only about N N of the domestic market will sell about N of its models this year twice as many as last year - it sees sales rising N N to N units next year - the company plans to expand plant capacity N N by N - by then it also hopes to begin producing a passenger car based on the N and selling for about $ N - hyundai and daewoo seem about the ssangyong threat but kia the auto maker is selling vehicles through its asia unit - it plans to sell N units in N - kia the only korean car maker that has seen its overseas sales grow in N aims at korea 's common man - its advantage has been the little pride sold as the ford in the u.s. - at N million won or $ N the is the car in south korea - along with two larger models the company claims N N of the domestic market - ford motor co. and japan 's mazda motor corp. have equity interests in kia - kia is the most aggressive of the korean big three in offering financing - loans for as long as five years make the cars very accessible with monthly payments as low as N won or $ N - daewoo motor a N joint venture with general motors corp. and the daewoo group conglomerate is the only auto maker that appears to be hurting - shipments of its to gm 's division are off about N N from a year ago a N N decline for hyundai and an N N increase for kia - moreover daewoo 's domestic sales have grown half as fast as sales of its rivals - the big problem for daewoo which holds about N N of the market is the long series of labor disruptions it suffered this year - but daewoo is expanding too - in fact a sister company daewoo shipbuilding and heavy machinery plans to build N by the mid-1990s - hyundai the korean market leader with a N N share also plans to jump into at the same time - it has a similar project for N cars a year - kia is reportedly also considering such a plan - even giant group is rumored in the korean press to be considering getting into the business a company spokesman had no comment - robert p. N years old was named president and chief administrative officer of this regional commercial bank - both posts had been vacant - robert N was named to the new positions of vice chairman and chief credit officer - many mutual fund investors picked up the phone yesterday but decided not to cash in their chips after all - as the stock market bounced back withdrawals of money from stock funds amounted to a mere compared with black monday when investors dumped $ N billion or about N N of assets - fidelity investments the nation 's largest fund company said phone volume was more than double its typical level but still half that of oct. N N - net outflows from fidelity 's stock funds stood at less than $ N million or below N N of the $ N billion cash position of the firm 's stock portfolios - much of the money was switched into the firm 's money market funds - outflows since the close of trading friday remain below one-third their level of two years ago fidelity said - other mutual fund companies reported even lighter withdrawal requests - and some investors at fidelity and elsewhere even began buying stock funds during the day - two years ago there was a lot of redemption activity and trouble with people getting through on the phone said head of the investment management division of the securities and exchange commission - this time we do n't have that at all - of course the relative calm could be jolted if the market again - and any strong surge in redemptions could force some funds to dump stocks to raise cash as some did during black monday - but funds generally are better prepared this time around - as a group their cash position of N N of assets in august the latest figure available is N N higher than two years earlier - many fund managers have boosted their cash levels in recent weeks - the biggest flurry of investor activity came early in the day - vanguard group inc. saw heavy exchanges from stock funds into money market funds after the telephone lines opened at N a.m - in the first hour the real nervous folks came along a spokesman said - but the pace of call volume in the first half-hour slowed considerably - at stevens & clark inc. phone calls came in at N N more than the normal pace through early afternoon - most of that increase came in the first hour after the phone lines opened at N a.m - as stocks rose in fact some investors changed course and reversed their sell orders - many funds allow investors to orders before the close of trading - at and at the smaller ivy funds group in mass. for instance some shareholders called early in the morning to switch money from stock funds to money market funds but later called back to reverse the switches - because mutual fund trades do n't take effect until the market close in this case at N p.m. these shareholders effectively stayed put - at fidelity 's office in downtown boston gerald sherman walked in shortly after N a.m. and placed an order to switch his retirement accounts out of three stock funds and into a money market fund - but by N p.m. with the market ahead for the day mr. sherman was preparing to undo his switch - it 's a nice feeling to know that things stabilized said mr. sherman the of a discount department store - but some investors continued to switch out of high-risk high-yield junk funds despite yesterday 's rebound from that market 's recent price declines - shareholders have been steadily out of several big junk funds the past several weeks as the $ N billion market was jolted by a cash crunch at campeau corp. and steadily declining prices - much of the money has been switched into money market funds fund executives say - instead of selling bonds to meet redemptions however some funds have borrowed from banks to meet withdrawal requests - this knocking down prices further - the $ N billion t. rowe price high yield fund was among the funds that borrowed during the campeau crisis says george j. collins president of t. rowe price associates inc - that way mr. collins says we did n't have to sell securities in a sloppy market - when the market stabilized he added the firm sold the bonds and quickly paid the loans back - tom contributed to this article - financial inc. said it agreed to acquire central of illinois inc. in a stock swap - shareholders of central a bank holding company based in sterling ill. will receive stock equal to N times central 's N earnings said - for the first nine months of N central earned $ N million - also a bank holding company has assets of $ N billion - central 's assets are $ N million - during its centennial year the wall street journal will report events of the past century that stand as milestones of american business history - soft contact lenses won federal blessing on march N N and quickly became eye for their makers - the food and drug administration that day said bausch & could start selling them in the u.s. - the product was more comfortable and less prone to falling out than hard contact lenses which had been around since N - bausch & sold the under a from national patent development which had gained the rights from the czechoslovakia academy of sciences - a invented them in N - the plastic lens itself over the eye while permitting to pass through - but the new lens became the eye of a storm - in september N california officials seized lenses made by companies after some showed of bacteria - in october doctors were the product 's safety some claiming it caused - and there were senate hearings on the questions in july N - the product the bad publicity and kept - the early soft lenses which cost $ N a set were expected to last for a year - in N extended wear versions designed to be for N days at a time offered - months ago a disposable seven-day model bowed a year 's supply costs about $ N - last month the fda and contact lens institute cautioned users that serious eye could result from wearing lenses more than seven days at a stretch - today N million of the N million americans using contact lenses are using the soft type - including the eye care products contacts account for $ N billion in annual retail sales - although bausch remains the leader among the six johnson & johnson with its new is coming on fast - the roller-coaster stock market is making life tougher for small companies trying to raise money - in the wake of friday 's plunge and yesterday 's rebound some companies are already deals and others wish they could - as in other jittery times many small businesses expect a particularly rough time raising funds as investors risky deals seeking safety in bigger companies - even if stock prices fully recover from friday 's sharp decline the unsettled conditions will many investors - the implication of an unsettled situation is that the thing could drop dramatically says henry jr. chairman of corp. a four-year-old biotechnology company that is planning a private placement of stock - the more that indicate risk the more the investor is going to drive a hard bargain - earlier this month inc. a mass. said it would accelerate expansion plans nationwide and offer more of its stock to the public - at the time its shares were selling above their initial offering price of $ N and bankers believed would sell new stock without a - but with the company 's shares standing at $ N yesterday a new offering seems unlikely company officials say - business however continues to be robust and the stock market has n't affected the concern 's expansion plans says a senior executive - other companies figure they ca n't avoid the market - we have capital requirements says mr. so we have to go ahead with a planned $ N billion private placement - unless the market goes right back up he says it may take us six to nine months to find the money instead of three - and the columbia md. company may have to settle for a lower price he adds - life is particularly for companies that had planned to go public this week - is becoming an investment-banking job requirement - robertson & co. a san francisco investment banking concern has a client that looked forward to making its initial public offering yesterday - officers of the company a health-care concern were very discouraged on friday and felt they should n't go public we felt they should says sanford robertson partner in the banking concern - as the market dropped friday robertson slashed the value of the offering by N N - yesterday when similar securities rebounded it the valuation up again - as of late yesterday the ipo was still on - for many the situation is especially discouraging because the market for was showing signs of strengthening after several years of weakness - we were just beginning to look at the increase in seeing the light at the end of the tunnel says frank jr. partner in funds a beverly hills calif. venture capital concern - but the tunnel 's just gotten longer - companies planning to go public are definitely taking a second look says allen senior analyst at the institute for research fort fla. which publishes the new issues newsletter on - he that the recent market slide translated into a N N to N N reduction in ipo proceeds to companies - many companies are - corp. had been planning to sell N N of its stock this week in an ipo that would raise up to $ N million - but now peter president says we 're making decisions on a day-to-day basis - and profitable the colo. concern could borrow funds if it decides against an ipo now he says - inc. an atlanta concern says it is still planning to go ahead with its ipo this week or next unless conditions change - it 's a situation right now says terry president - delayed financings also would affect the operations of many companies - sierra tucson cos. a tucson ariz. operator of centers has a planned doubling of capacity riding on an ipo scheduled for next week - william president says he still thinks the ipo will succeed - if it does n't he says the company would have to change its expansion timetable - but the market turmoil could be partially beneficial for some small businesses - in a sagging market the federal reserve system might flood the market with funds and that should bring interest rates down says leonard t. vice president of the bank of new england boston - james g. president of savings bank mass. says the market turmoil is an for small business - for small companies he says interest rates are far more important than what happens on stock exchanges - mr. thinks rates are heading down helping small companies - peter biotechnology analyst for securities international chicago thinks market uncertainty may encourage small companies to form more strategic alliances with big corporations - partly because the N market crash made it harder for them to find financing many high-technology concerns have made such alliances recently - some even see a silver in the dark clouds - alan wells president of wells & co. a new york merger specialist thinks investors may lose their enthusiasm for leveraged buy-out and giant takeover deals - instead they could turn to investing in smaller deals involving smaller companies he says - and william e. jr. a university of new hampshire management professor and director of venture capital network inc. says the market 's gyrations will the investors ' lack of control in big stock investments - this will add to the appeal of small business he says where investors often have a degree of influence - bay financial corp. hurt by high debts and deteriorating real estate investments reported a wider loss for the fourth quarter and said it might be forced to seek a bankruptcy-court reorganization if it ca n't its borrowings - bay said a substantial part of its debt outstanding is in default as a result of inability to sell certain properties quickly and lower-than-expected prices for sales made - the company said its real estate portfolio is highly leveraged while about two-thirds of its investments are n't - thus it is coming up short on a big bet that quick sales at higher prices would enable it to keep up with mortgage and other debt payments - according to its latest annual report about a quarter of the company 's holdings are in massachusetts in the midst of a real-estate slump - the company said it had a net loss in its fourth quarter ended june N of $ N million or $ N a share on revenue of $ N million - a year earlier the company had a loss of $ N million or $ N a share on revenue of $ N million - for the year it had a net loss of $ N million or $ N a share on revenue of $ N million - in the previous year it had a loss of $ N million or $ N a share on revenue of $ N million - although it is having serious problems bay said the value of its holdings minus debt was equal to $ N a share at june N based on a recent - book value per share which is based on investments at cost was a negative $ N a share - a year earlier value per share was $ N and book value was $ N a share - annualized interest rates on certain investments as reported by the federal reserve board on a basis N and wednesday october N N - adjusted for constant maturity - inc. reported a N N decline in third-quarter net income but the company said that excluding unusual gains in both quarters operating profit rose N N - the electronics automotive and aerospace concern said third-quarter net was $ N million or N cents a share down from $ N million or $ N a share a year earlier - share earnings are reported on a fully diluted basis by company tradition - results for the N quarter included a gain of $ N a share from sale of the pump and cable units partly offset by a charge of N cents a share for recall of truck steering systems - the latest quarter included a gain of N cents a share as a partial reversal of the recall charge because the reserve established last year exceeded the actual recall costs - sales for the quarter rose N N to $ N billion from $ N billion with all three major product groups reporting gains - the company said aerospace and defense sales were up N N for the quarter to $ N million and operating profit climbed N N to $ N million mainly because of improved program performance in spacecraft and contracts - automotive sales jumped N N to $ N million mainly because of higher sales of air bags and other passenger restraint systems said - the group had an operating profit of $ N million against a loss of $ N million a year earlier - however excluding the year-earlier charge for recall of steering gear operating profit in the latest quarter declined N N reflecting higher start-up and product development expenses in systems - materials and production costs also rose said - the information systems segment had a N N jump sales to $ N million - an acquisition accounted for half the sales rise said - operating profit rose to $ N million from $ N million - for the nine months 's net was $ N million or $ N a share down N N from $ N million or $ N a share a year earlier - sales rose N N to $ N billion from $ N billion - a by an not though english butler in his proceeds as if the realistic english novel of like herself still ruled the waves - in fact 's the remains of the day N pages $ N is both an to traditional english forms and a dramatic of them - it implies that the british empire was rooted in its subjects ' minds and and argues that its flaws were in the defensive willful and especially the of its domestic - as the stevens the butler of hall over such terms as dignity service and loyalty we see how the soul - stevens 's of the public and private like his master 's all it was designed to preserve - such the - the cuts to the quick - it 's N the year the suez crisis marked the final end of empire - as he stands on a hill at the beginning of a motor from to where a former perhaps the victim of an unhappy 20-year marriage perhaps he hopes with more than he will ever acknowledge not to return to domestic service stevens surveys the view and thereby provides a a and the author 's for the of the novel we 're reading - we call this land of great britain and there may be those who believe this a somewhat practice - yet i would venture that the landscape of our country alone would justify the use of this - it is the very lack of obvious drama or that sets the beauty of our land apart - what is is the of that beauty its sense of restraint - it is as though the land knows of its own beauty of its own and feels no need to it - in comparison the sorts of sights offered in such places as africa and america though undoubtedly very exciting would i am sure strike the objective as on account of their - an landscape - an mountain - but let stevens continue in his comic manner his efforts at always fail most this whole question is very to the question that has caused much debate in our profession over the years what is a great butler - his answer is one of a dignity in keeping with his position - such dignity has to do with a butler 's ability not to abandon the professional being he - he will not be shaken out by external events however surprising or - are unable to be because they are as a breed of the emotional restraint which only the english race are capable of - despite his racial advantage to be a great butler is a calling one 's is not unlike general 's headquarters during a battle - if for example in the midst of a great social occasion such as an international conference on the treaty in N one 's father himself a great butler once should happen to die of a one must continue to serve the port please do n't think me improper in not to see my father in his condition just at this moment - you see i know my father would have me to carry on just now - it is this kind of dignity and restraint that allows stevens to declare for all its sad associations whenever i recall that evening today i find i do so with a large sense of - we note the imperial public word used to deny private rage and - that stevens himself is not or but funny and sad and is entirely the author 's - mr. 's ability to create a voice that permits him to explore such domestic cultural and political themes was clear in his previous novel an artist of the floating world set in japan after the war - now shifting his scene from the country he left at five to the england he has lived in for nearly N years he has a novel in the mode of henry james and - with great he considers not only and utterly sexual love but british the 's with democracy and support of and the moral of loyalty it is in practice simply not possible to adopt such a critical attitude an employer and at the same time provide good service - this employer all that i find noble and - i will devote myself to serving him - this is loyalty - in the end after meeting with the former stevens sits by the at thinking of her and of his employer and declares i trusted - i trusted in his 's wisdom - i ca n't even say i made my own mistakes - really one has to ask what dignity is there in that - the loyal has come full circle - what is - what is dignity - we understand such wisdom must be the of only spreads her wings at - but as the remains of the day so demonstrates with quiet such wisdom can be in art - mr. teaches english and literature at columbia university - corp. said its subsidiary completed the previously announced sale of its air separation plant and related assets in wis. to aga gas inc. cleveland - the price was n't disclosed - the transaction is part of 's continuing program to shed 's industrial gas interests and expand the subsidiary 's propane business - since june has more than $ N million from industrial gas and reinvested more than $ N million to acquire three propane distributors - is a gas and electric utility and distributes propane nationally through its subsidiary - who represents the soviet airline aeroflot here has some that are wild even by the current standards of perestroika - in his office the runway of shannon airport mr. throws out what he calls just ideas - first he suggests group ltd. the international aircraft leasing company based in ireland could lease some of its boeing to the soviet airline - then aer the irish flag carrier could teach aeroflot pilots to fly the and the fleet could be based here at shannon airport - that 's not all he says - aer the irish airport authority could build a cargo terminal in the soviet union - aeroflot could lease some of its cargo planes to aer through for a joint-venture cargo airline - and then there is his notion of an charter airline to ferry to los angeles via shannon - have the freedoms of glasnost gone to mr. 's head - hardly - the aviation connection is alive and well here at shannon airport - is indeed talking about leasing western planes to aeroflot and even about buying - aer is in discussions with the soviet carrier about a cargo venture and other possibilities - aer already has so many ventures with aeroflot that its chief executive is studying russian - unlikely as it may seem tiny politically neutral ireland has the mighty soviet airline bureaucracy - and as aeroflot struggles to boost its service standards upgrade its fleet and pursue commercial opportunities the irish aviation industry seems poised to benefit - irish and soviet people are similar says mr. - they look the same - they 're very friendly - moreover he says irish companies are small but - we have to study their experience very well he says - we must find any way to get business - the two groups have been working together since the late 1970s long before soviet joint ventures were the rage in the west - aeroflot carried about N million passengers last year and shannon airport the airline 's largest transit airport outside the soviet union saw N aeroflot flights and N passengers pass through - an apartment complex down the road is the and staging area for more than N aeroflot pilots and flight attendants - the airport 's biggest supplier of aircraft fuel is the soviet union - from the port of each year unload N million gallons of fuel into a special tank farm at the airport - what aeroflot does n't pour into its own is to the airport authority which it to N western carriers including air france trans world airlines and pakistan international airlines - aeroflot thus pays its landing fees and bills with fuel preserving its hard currency - that is n't all - last year the irish airport authority in a joint venture with aeroflot opened four duty-free shops at moscow 's airport - aer now manages duty-free sales on all aeroflot international flights out of moscow - duty-free shops in 's airport opened in july and shops in hotels and on the are coming soon - aer is talking about similar joint ventures in and in a black sea resort and even has a project cooking with the city of - aeroflot 's international fleet of N planes is being and at shannon airport - thanks to a new agreement and the ability of irish travel agents to issue aeroflot tickets tourists here are taking advantage of aeroflot 's reasonable prices to board flights in shannon for holidays in and mexico city - the fare to is N irish punts $ N - jamaica costs N punts - a formal blessing of sorts was on this friendship in april when mikhail and gorbachev stopped here for talks with irish prime minister charles - new trade accords were signed - it all started with geography - when it opened in N shannon was the first in europe for airplanes flying from north america - advances in aircraft fuel efficiency over the years made a shannon stop unnecessary for most western air fleets but aeroflot still flies inefficient that ca n't make it from moscow to managua on one - as a result ireland did n't the soviets after they shot down a korean air lines jetliner over the sea of japan in N though it suspended direct flights for two months - in fact aer started russians from shannon to new york when washington stripped aeroflot of its u.s. landing rights - today aer is making a of money from its soviet friendship - and with those contacts in place it could be relatively simple to add aer and to the team - then perhaps mr. 's ideas would n't sound like so much - britain 's industrial production rose N N in august from july and was up N N from august N according to provisional data from the central statistical office - output in the energy sector which can vary greatly with swings in the oil market rose N N in august from may but was down N N from a year earlier - the latest figures compare with july 's N N rise and N N year-to-year fall - when corp. begins shipping steel from the world 's first plant this month it will begin testing the competitive of its giant competitors - the new technology which creates a very thin piece of steel reduces the costs of making flat-rolled sheets - an kenneth iverson 's chairman says the company 's plant eventually will make a ton of steel in N man hours compared with four to six man hours at a conventional mill - we 've had the russians and chinese and people from india visiting us mr. iverson - everyone in the world is watching us very closely - especially his neighbors the major u.s. steelmakers - already usx corp. and armco inc. are studying 's technology to see if they can adopt it - says the chief executive officer of a major midwest steel company it 's damn worrisome - the steel industry is about to be turned by a 1990s technology revolution - new efficient and sophisticated processes make it easier for smaller less companies to make steel at a fraction of what big steel paid decades ago - it also enables minimills finally to get a in the flat-rolled steel market the major steelmakers ' largest most and until now market - but such technology is only the beginning - eager engineers and direct casting which by the end of the 1990s will enable production without coke and blast - those massive structures while cost and environmental headaches effectively locked out all but giants from - there 's a revolution ahead of us that will ultimately change the way we market and distribute steel says william dennis vice president manufacturing and technology for the american iron and steel institute - it is n't that major steelmakers have ignored high technology - in fact they 've spent billions of dollars to boost the percentage of cast steel to N N in N from N N five years before - moreover their balance sheets are rich with diversity their old plants and work forces lean - but that wo n't - it 's no longer enough to beat the guy down the street - you have to beat everyone around the world says mr. dennis - he wants to see steelmakers more involved in computers and intelligence - the problem they 're with huge plants that require costly maintenance - and try new dollars free in a market that is softening hurt by a strong dollar and concerned about overcapacity the industry 's - the technology revolution is going to be very threatening to established producers says peter marcus an analyst with painewebber inc - they 've got too much invested in the old stuff and they ca n't get their workers to be flexible - no one expects minimills to major integrated steelmakers who remain the of steel used for autos and refrigerators - 's plant in ind. ultimately will produce only one million tons annually a drop in the flat-rolled steel and it will be years before such plants can compete in the market - still flat-rolled is the steel industry 's bread and butter representing about half of the N million tons of steel expected to be shipped this year - moreover the process is n't without its headaches - because all operations are connected one equipment failure forces a complete plant shutdown - on some days the plant does n't produce anything - at this point the capacity wo n't make a great in the integrated market but it does challenge them to develop new markets says james mccall vice president materials at a technology and giant based in columbus ohio - indeed with demand for steel not growing fast enough to absorb capacity steelmakers will have to change the way they do business - in the past says armco 's chief economist john steelmakers made a product and set it out on the - we said we 've got a product if you want it you can buy it he says adding now we 're figuring out what people need and are going back to make it - armco 's sales representatives visit the general motors corp. 's assembly plant in kansas city mo. two or three days a week - when they determined that gm needed parts more quickly armco convinced a steel service center to build a processing plant nearby so shipments could be delivered within N minutes - such relationships with major clients car and makers is a means of survival especially when those key clients are relying on a smaller pool of producers and with plastic and aluminum makers - for example when detroit began talking about cars the american iron and steel institute began a major lobbying effort to show auto makers how they could use steel more efficiently by simply how a car door is assembled - but steelmakers must also find new markets - after letting take the recycling lead a group of the nation 's largest steelmakers started a recycling institute to promote steel cans to an environmentally nation - 's mr. mccall thinks steelmakers should concentrate more on construction - weirton steel corp. weirton w. va. for example is touting to homeowners fashionable steel doors with glass as a secure and alternative to wooden or aluminum ones - other steelmakers steel covering - still others are looking at overseas markets - usx is drilling pipe to soviet union - this year the nation 's largest steelmaker its overseas sales operation - producers also are trying to by concentrating on output such as coated and products which remain beyond the reach of minimills - almost all programs announced by major steelmakers within the past year involve building lines used to produce steel for such products as household appliances and car doors - but unfortunately that segment is much smaller than the bread-and-butter flat-rolled steel - it 's like everyone climbing out of the ii and getting into a says john jacobson an analyst with consultants - after a while someone has to go over the side - although he does n't expect any he does see more plants being sold or closed - robert crandall with the institute agrees - unless there is an enormous rate of economic growth or a further drop in the dollar it 's unlikely that consumption of u.s. produced steel will grow sufficiently to offset the growth of minimills - not to mention the of imports - japanese and european steelmakers which have led the recent technology developments are awaiting the lifting of trade restraints in N - moreover the u.s. can expect more competition from low-cost producing pacific and latin american countries - a taiwanese steelmaker recently announced plans to build a plant - people think of the steel business as an old and mundane business says mr. iverson - they 're dead wrong - \* usx ltv bethlehem inland armco national steel - \*\* projected - polaroid corp. 's damages case against eastman kodak co. one of the highest stakes corporate trials ever is getting attention on wall street - after N days of testimony in federal court in boston the trial is being all but ignored by analysts and patent attorneys - most have read the pre-trial documents however and estimate kodak will be ordered to pay $ N billion to $ N billion for on seven polaroid patents - that may be the largest patent award ever but it is well below the $ N billion polaroid seeks - the highest patent damage award to date was in N when smith international inc. was ordered to pay $ N million to baker hughes inc. for on a patent on an oil drilling bit seal - the two companies later agreed to settle for $ N million - few analysts think it is worth their time to through the polaroid trial testimony - it 's like for gold outside of grand central station - you might find something but the chances are low said michael an analyst at wertheim schroder & co - and eugene glazer an analyst at dean witter reynolds inc. said if you hired an attorney to be there all the time and give you a prediction of the eventual award i would be willing to bet that he would be off by a lot - a trial in the early 1980s determined that kodak based in rochester n.y. infringed on patents of polaroid of cambridge mass - the main issues remaining are how to calculate damages and whether the infringement was willful and - if so the damages could be tripled - two analysts who have read the david nelson of shearson lehman hutton inc. and d. a litigation analyst at simpson & co. think judge a. david will decide in kodak 's favor on the willful and issue - mr. said testimony by kodak 's patent counsel francis t. carr of & showed that he worked with kodak from the outset of the project in an effort to avoid infringement - carr told kodak on many occasions to avoid various features because of polaroid 's patent positions and kodak followed his advice in every instance mr. said - but irving a patent expert at george mason university school of law who is familiar with the case said the fact that seven patents were infringed suggests that infringement was willful - it 's difficult to be that consistently wrong - observers also wonder whether judge will use the method of determining damages which polaroid favors because it would result in a larger award or the reasonable royalty method - polaroid claims it could have manufactured and sold all the instant cameras and film sold by kodak if kodak had n't entered the market - moreover polaroid contends it could have sold them at a higher price and thus made higher profits because it would n't have been forced to match kodak 's lower prices - each side has called a harvard business school professor to testify on that issue - kodak hired robert and polaroid brought in robert j. - there 's nothing that says that people at harvard business school have to agree with each other said mr. - testimony is expected to continue until early december - a decision is n't expected until some time next year - international business machines corp. said earnings tumbled N N in the third quarter even a bit further than expected the outlook doubtful for the next few quarters - the main reason was a delay in shipment of new high-end disk drives a business that accounts for some N N of ibm 's $ N billion of annual revenue - ibm which the poor results three weeks ago also cited an increase in its leasing business which tends to lock in business long-term but cut revenue in the near term - in addition ibm noted that the stronger dollar has cut the value of overseas revenue and earnings when they are translated into dollars - earnings fell to $ N million or $ N a share somewhat below securities analysts ' revised expectations of around $ N a share - that compared with the year-earlier $ N billion or $ N a share which was inflated by a gain from the sale of some mci communications corp. stock and by an unspecified amount from a payment by fujitsu ltd. relating to a software dispute - revenue climbed N N to $ N billion from $ N billion - ibm armonk n.y. remained upbeat - the computer giant whose u.s. results have been dismal for years noted that revenue rose again in the u.s. in the third quarter following an increase in the second period - the company said in a statement that demand for ibm products and services continues to be good world-wide - we do not see anything in the fundamentals of our business that would cause us to change our strategy of investing for profitable growth - securities analysts however remained - i think N will be another year said steve of first boston - jay stevens of dean witter actually cut his per-share earnings estimate to $ N from $ N for N and to $ N from $ N in N because he decided sales would be even weaker than he had expected - both estimates would mark declines from the N net of $ N billion or $ N a share which itself was well below the record ibm set in N - mr. stevens said he kept a recommendation on the stock only because all the damage has been done - he said the stock has n't traded below N N times book value over the past N years which at the moment to a stock price of $ N - the stock closed yesterday at $ N a share up just $ N in composite trading on the new york stock exchange as the market surged - analysts worry that the disk-drive and leasing problems will last at least through the first quarter - a key part of the question is how soon does this disk-drive come and how soon does production up said steve cohen at financial group - and the input i 've had from customers is that it still could be a while - on leasing bob at research said he thinks ibm has hurt itself - he said ibm has priced its leases aggressively thinking that would help win business - but he said ibm would have won the business anyway as a sale to a third party that would have then leased the equipment to the customer - he said ibm has not only hurt its short-term revenue outlook but has also been losing money on its leases - bob executive vice president of marketing at inc. a huge leasing firm said to put it mildly ibm credit has been doing some of the worst economic deals of any leasing company we have ever seen - ibm is expected to get a boost soon when it some new versions of its mainframes - but the basic technology in the line is almost five years old which means it is long in the and competitors are rolling out strong products of their own - ibm is gaining momentum in the personal-computer market and is expected to introduce some impressive workstations early next year - but it 's hard to squeeze much profit out of the personal-computer business these days and the workstation market while important is too small to rely on for much growth - the disk drives will sell well when they finally become available - but the ibm 's highly successful line is losing its momentum and some analysts said sales could even decline in the fourth quarter - in addition ibm 's growth in software in the third quarter was just N N well below historical levels even when adjusted to reflect last year 's payment from fujitsu and the stronger dollar - and expenses up N N in the quarter have stayed high - in the nine months ibm earned $ N billion or $ N a share down N N from the year-earlier $ N billion or $ N a share - revenue increased N N to $ N billion from $ N billion - pepsico inc. 's chairman said he is more than comfortable with analysts ' estimates that third-quarter earnings rose to at least N cents to $ N a share from N cents the year earlier - d. wayne calloway also chief executive officer of the company indicated that he expects analysts to raise their forecasts for N after the company releases its earnings today - so far analysts have said they are looking for $ N to $ N a share - after today 's announcement that range could increase to $ N to $ N a share - the official said he also would be comfortable with that new range - in N the soft-drink giant earned $ N a share - results for N will include about N cents a share from the effects of snack-food and bottling company acquisitions - in composite trading on the new york stock exchange the company closed yesterday at $ N a share up $ N - the company said third-quarter sales are expected to increase N N from $ N billion of last year 's third quarter - domestic soft-drink case sales are estimated to have risen only N N in the third quarter well below the N N to N N growth of recent years but about in line with the rest of the soft-drink industry - mr. calloway blamed the slower volume on weather a of new products in the industry and to a much lesser extent pricing - pepsico said its soft-drink prices were about N N higher in the quarter - mr. calloway also noted that soft-drink volume rose a hefty N N in last year 's third quarter making the comparison more difficult - international soft-drink volume was up about N N - snack-food increased a strong N N in the third quarter while domestic profit increased in double mr. calloway said - excluding the british snack-food business acquired in july snack-food international jumped N N with sales strong in spain mexico and brazil - total snack-food profit rose N N - led by pizza hut and bell restaurant earnings increased about N N in the third quarter on a N N sales increase - sales for pizza hut rose about N N while bell 's increased N N as the chain continues to benefit from its strategy - bell has turned around declining customer counts by permanently lowering the price of its - same for kentucky fried chicken which has struggled with increased competition in the fast-food chicken market and a lack of new products rose only N N - the operation which has been slow to respond to consumers ' shifting away from fried foods has been developing a product that may be introduced nationally at the end of next year - the new product has performed well in a market test in las vegas nev. mr. calloway said - after a four-year $ N billion acquisition binge that brought a major soft-drink company soda a fast-food chain and an overseas snack-food giant to pepsi mr. calloway said he does n't expect any major acquisition in the next year or so - but you never can tell he added you have to take advantage of opportunities - president bush chose martin a longtime friend from texas to be chairman of the federal energy regulatory commission - mr. would succeed who is resigning - the white house said ms. a chicago who previously held posts at the energy department and ferc is leaving to become a vice president of first chicago corp - mr. an attorney in midland texas has been at the interior department - he met mr. bush in the 1950s when the president was a young oil man in midland and mr. was a lawyer for an oil firm - the ferc is a commission that billions of dollars of interstate wholesale energy transactions - mr. 's appointment is subject to confirmation by the senate - administration officials said a date for ms. 's departure has n't been set - california real estate investment corp. said its directors declared a dividend of five cents per class a common stock payable nov. N to stock of record oct. N - the dividend represents the balance of its regular quarterly payout of N cents a share of which half was paid july N in a final distribution prior to its merger with real estate investment corp. also in july - the company said it hopes to resume its schedule of regular quarterly dividends at the end of this year - hydro-quebec said it notified central maine power co. it will cancel a $ N billion contract to supply electricity to the maine utility - the owned utility said it is up the deal because the contract 's objectives ca n't be - hydro-quebec said maine regulators ' refusal to approve the contract earlier this year halted work on transmission lines and stopped negotiations for resale of electricity carried through maine to other utilities - it would now be impossible to begin deliveries in N a hydro-quebec official said - the contract was to run from N to N - under the contract hydro-quebec was to supply N of power to central maine power starting in N N starting in N and N starting in - hydro-quebec said maine regulators ' refusal to approve the contract means central maine power has lost its place in line - we wo n't sign any new contracts with deliveries beginning earlier than N the hydro-quebec official said - he said hydro-quebec already has some customers in mind for the power that was to be delivered to maine - nothing has happened since we signed the contract to undermine our conviction that hydro-quebec was the most environmentally acceptable choice for meeting a part of our customers ' energy needs through the year N said central maine senior vice president donald f. kelly - central maine said it is evaluating many energy options to make up for the lost future power including new energy generation and management proposals from new england and possibly new canadian purchases - chicago options traders were among the big victims of friday 's plunging stock market including one small firm that required an emergency $ N million bailout - while monday 's rebounding markets helped other investors recoup losses many options customers and professional traders in stock-index options and the options on takeover stocks were left with multimillion-dollar losses traders here and in new york said - options traders were hurt worse than others on friday because of the highly volatile nature of options which often rise or fall in value several times the amount of the price change in the individual stock or index of stocks on which they are based - thus options traders friday were stuck with losses that also were several times larger than those suffered by many stock traders in new york - jeffrey miller of miller & co. said that given the high degree of leverage in the options market it is very easy for these guys to get wiped out - that may just be the nature of these highly leveraged little creatures - an options contract gives the holder the right to buy call or sell put a specific amount of stock or in this case the value of a stock index based on a price within a given time period - options traders who in return for a small fee or premium had previously sold put options on stocks or stock indexes were forced on friday to buy those contracts back at the previously agreed prices which were substantially above those in the market as it was falling - they then had no choice in many cases but to sell the contracts at prevailing prices in most cases at a substantial loss - the latest round of losses is likely to be a serious blow to the chicago board options exchange which has never fully recovered from the of black monday when investors fled the market because of huge losses - making matters worse was the fact that late friday afternoon the cboe halted stock-index options trading in step with the chicago mercantile exchange 's halt in stock-index futures - but while the merc reopened a half hour later the cboe remained closed leaving many options traders unable to make trades that might have reduced the losses - cboe chairman duke said that unlike the futures market the options exchange has to open in a that allows each different options series to trade - exchange officials that they would n't have been able to make such a with the time remaining friday afternoon and with the stock-index futures on the verge of closing for a second and final time the cboe that its best course was to remain closed - the damage was so bad at fossett corp. an options trading firm here that it was forced to transfer its accounts to first options of chicago a unit of continental bank corp. as a result of options trading losses - so far is the only member of a financial exchange to be forced to be taken over by another firm as a result of friday 's rout - fossett still had several million dollars in capital left after friday 's close of trading but not enough that regulators worried about another potential market plunge yesterday would let it reopen for trading options exchange officials said - thus in an unprecedented arrangement the of the transfer the cboe the american stock exchange and the options clearing corp. as well as the firm 's owner stephen fossett put up a total of $ N million to guarantee the customer positions being transferred to the bank holding company subsidiary in case the market plunged again yesterday - s. iii vice chairman of continental bank first options ' parent company said the firm took on about N accounts formerly held by fossett almost all of them to professional floor traders - steve and his firm were still worth a lot of money mr. said - a package of credit support was put together including the assets of steve and his firm - the bailout was together over the weekend with officials from the federal reserve board securities and exchange commission comptroller of the currency and treasury as well as the options exchanges - it was great to have the luxury of time mr. said - at one point an options industry official had to talk the federal reserve bank of chicago 's night into giving him the home phone number of chicago fed president - first options did n't have to put any money into the bailout - yesterday 's rally in the stock futures and options markets led cboe and amex officials to conclude that the $ N million in guarantees almost certainly wo n't need to be tapped by first options - the fossett firm had some losses and liquidity problems during the october N crash as well mr. said - a federal official said that continental bank worked with securities and banking regulators over the weekend to fashion the fossett bailout but that conditions were n't by those agencies - it was their business decision the official said - officials at options clearing corp. which processes all options trades for u.s. exchanges said that the $ N million guarantee was unprecedented but was necessary to help insure the integrity of the options markets - it was an extraordinary situation that needed extraordinary steps said paul stevens president and chief operating officer - mr. stevens declined to give the specific contributions to the $ N million guarantee from each participant - but cboe and amex officials said that options clearing corp. contributed $ N million to the guarantee the cboe put up $ N million the amex added $ N million and $ N million came from mr. fossett 's own assets - mr. fossett could n't be reached to comment - foster takes off her herself on a chair and gently forward - with a tape playing in the background the hands of begin to work on ms. foster 's neck and - it 's like an in this room ms. foster - the room in question is the directors ' of co. N floors above the of pittsburgh - there amid oil paintings and marble tables massages are every wednesday - on days that i 'm really busy says ms. foster who works in public relations for the company it seems to take time off for a massage - although such sessions may never replace coffee breaks on-site massage as it is known in the trade is certainly corporate america - in some companies middle managers massage into the office fearful that executives wo n't approve - ms. foster 's is nothing like the enjoyed by visitors - nor does it at all resemble despite what some executives think the more intimate variety offered at specialty in bad parts of town - on the contrary office usually take place in conference rooms where employees relax in specially designed chairs fully - the massages last N minutes and typically cost about $ N - some companies including even pay part of the fee - ms. has been seeing some N clients a visit since the program was started at last year - anthony the company 's chairman by her firm touch saying regular massages are a for his old football injuries - massage advocates say that the head neck and back can go a long way toward easing tension and improving morale - they also insist that is a basic need as powerful as the need for food or sleep and that the office is as good a place as any to do it - the blood flows to your head you feel and you do n't feel tension around the head or neck says an operations supervisor at the social security office in grand mich. where massages began last month - when you leave the room after your massage people say you look like you 're - adds the who her trade in the grand office they fall in love with my hands - not everyone however is at ease with office massage - three years ago the internal revenue service 's office in san jose calif. opened its doors to on-site massage - and even though employees paid the bill taxpayers - sometimes with the release of stress you hear and coming out of the room explains morgan banks the agency 's health specialist - and you ca n't have taxpayers coming into an audit hearing and - last month the complaints and the massages ended - now we 're looking for a room with walls ms. banks says - massage also has an image problem to contend with - some have tried to get around this by calling themselves and describing their office visits as breaks - but massage no matter how is still associated in many minds with fronts for and that makes some executives nervous - last year the research and development division of weyerhaeuser co. the large concern invited a to its wash. offices - phil a software engineer was an eager customer - you build up a lot of tension working at a terminal all day he says - but after about eight months the vice president of the division ed learned about the sessions and brought them to a halt - mr. says his only beef was that the massages were being given in a company conference room the department 's health facility would have been fine - in my view massages should be managed with an appropriate of males and around he says - given such attitudes some corporate prefer to go about their business quietly - russell of park n.j. says he has been working for the past year at a huge chemical and manufacturing concern in new york to the company 's executives - he visits the same department every two or three weeks - his massage chair is kept in a and a secretary him past security - this is common with a lot of large companies says mr. who worked for american telephone & telegraph co. for N years before choosing his current trade - managers he contends are afraid how they 're going to look in the eyes of their - my vision is to change human touch - my attitude is let 's come out of the - occasionally all that 's needed is a little - a st. louis won over officials at emerson electric co. a maker of electrical and electronic equipment by providing documents and other articles the benefits of massage - she notes that she also during her weekly visits - i pull my hair back wear a little makeup and look corporate says ms. who has been visiting emerson since january - if i go in there as i normally dress they 'd ask who is this - the father of on-site massage is david palmer a san francisco whose mission is to save the - to help do this mr. palmer developed a portable massage chair three years ago that he hopes will bring structured into mainstream america - the culture is not ready to take off its clothes lie down and be touched for an hour for $ N he says - the idea is to keep the clothes on and to keep people - the chair is a way to package massage - sitting in one of mr. palmer 's chairs which cost $ N and have since been by others is a bit like a - customers lean forward rest their on side supports and their face in on the back of the chair - ms. the grand says she has heard the compared to something out of the spanish - mr. palmer who serves as president of the on-site massage association and writes an industry newsletter says some N practitioners out of about N certified across the country now use massage chairs in the workplace as well as on street corners in airports and and at and other where people can be found - a in colo. had a scary experience while a man in a supermarket as part of a store promotion - three minutes into the massage the man up began shaking and turned red - were called - a week later the man told mr. he had suffered a mild heart attack unrelated to the massage - it was a powerful point in my career says the mr. who has since taken out a $ N million liability policy for his business - but he pulled through and after the left there were still six people in line waiting for a massage - the next woman was older and i was afraid to touch her - but it 's like falling off a horse and getting back on - despite the number of fans that office massage has won some look down on it arguing that naked are the only way to go - linda who does work in pittsburgh says that while on-site massage is better than nothing tired workers should realize it is only the tip of the - whole areas of their bodies are neglected she says adding that clothes the experience - there 's nothing like skin to skin - in what is believed to be the first cancellation of a loan to china since the june N killings in beijing an international bank syndicate has terminated a $ N million credit for a shanghai property project - the syndicate led by asia ltd. agreed last november to provide the loan to asia development corp. a u.s. property developer - but several weeks ago in the wake of the beijing killings the loan was canceled according to bankers and executives close to the project - asia development and declined to comment on the move - lenders had doubts about the project even before june N but the harsh crackdown which caused many businesses to their china transactions gave the banks the out they wanted says an official close to the shanghai venture - the decision to cancel the loan the tough attitude bankers have taken toward china since june N - while some commercial lending has resumed international lenders remain nervous about china 's economic troubles and foreign debt $ N billion at the end of N - many loans are being especially those tied to the hotel sector which has been hit hard by a N tourism slump - many bankers view loans as particularly risky - the canceled shanghai loan leaves asia development a small concern with a apartment building and heavy debts - the company owes $ N million to the on group the project 's hong kong contractor and a significant though unspecified amount in legal fees to brothers a u.s. law firm the sources say - the project known as lotus mansion has been mired in controversy - when the loan agreement was announced it was hailed as one of the first western-style financing transactions ever used in china - unlike most loans to china there was no chinese - instead the banks secured a promise from state-owned bank of communications that it would lend asia development the entire $ N million at maturity to finance repayment of the original borrowing - the loan was to have in just two to three years as soon as construction was completed - but in a letter sent in august to asia development said the loan was terminated because the developer had failed to deliver adequate financial data and pay certain fees to the committee on time according to officials close to the project - creditors involved in the project contend however that the termination actually had nothing to do with these technical violations - instead the creditors say the loan fell victim to nervousness about china 's political turmoil as well as to concern about the loan 's security - the bank syndicate is made up mostly of european banks but it includes china 's state-owned industrial bank - the N banks in the syndicate sustained no monetary losses because none of the credit facility had been drawn down - k mart corp. agreed to acquire pace membership warehouse inc. for $ N a share or $ N million in a move to expand its presence in the rapidly growing business - the proposed merger comes as k mart 's profit is declining and sales at its core discount stores are rising more slowly than at such competitors as stores inc - k mart based in mich. recently said net income would fall for the third consecutive quarter after a N N drop in the first half of its current fiscal year - the membership concept has great potential the company 's chairman joseph e. said in a statement - warehouse clubs typically carry general merchandise and food products which they sell for close to wholesale prices in stores - shoppers many of whom operate small businesses pay annual membership fees which provide an income base for the stores - k mart tested the sector last year with its acquisition of a N N interest in inc - but the chain which operates as a joint venture between k mart and shv holdings n.v. of the netherlands has only six stores and annual sales that one analyst estimated at about $ N million - pace based in colo. operates N stores - the company had losses for several years before turning profitable in fiscal N - in the year ended jan. N pace up profit of $ N million or N cents a share after a tax-loss carry-forward on sales of $ N billion and analysts expect its results to continue to improve - the company turned the corner fairly recently in profitability said of painewebber inc. who had been forecasting a N N jump in pace 's net income from operations this year and another N N increase next year - warehouse productivity is really beginning to take off - but some analysts contend k mart has agreed to pay too much for pace - even if you look at it as a turnaround situation it 's expensive said wayne of prudential-bache securities inc - in my opinion you would only pay that kind of price if you were getting a premier player in the industry - ms. of painewebber raised a more fundamental question about the deal - if k mart ca n't get its act together in discounting why is it spending time worrying about other growing markets - she said i would say k mart 's number one job is to address its market-share loss in discount stores which longer-term will lead to improved profit margins - at that point perhaps diversification would be appropriate - but k mart 's mr. is intent on pushing the company into new retail businesses - for instance k mart is opening big food and general merchandise stores called and stores specializing in office products and sporting goods - it also operates pay less drug stores and builders square home improvement stores - in composite trading on the new york stock exchange k mart closed yesterday at $ N a share up N cents - pace rose $ N to close at $ N a share in national over-the-counter trading - a k mart spokesman said the acquisition would be financed with short-term borrowings - under terms of the agreement a k mart subsidiary will soon make a tender offer for pace shares - among the conditions of the offer is that pace shareholders tender a majority of the company 's shares outstanding - the companies said pace would ill continue to operate under its present management - g. william president of stations was named chief executive officer of the unit of this media company effective jan. N - he will succeed joel who will remain a vice president of the company and continue to represent stations in several industry organizations the company said - literally - traders nervously watching their quotron machines yesterday morning were stunned to see the dow jones industrial average plummet N points in seconds - a minute later it soared N points then back down N points N below friday 's close - it was crazy said neil general partner of capital corp - it was like flying without a pilot in the front of the plane - but those who said this ca n't be happening were right - the were wrong - quotron systems inc. a citicorp unit blamed the on a timing problem in our software caused by the enormous early volume about N million shares in the first hour of new york stock exchange trading - the prices of the individual stocks that make up the average were correct quotron said but the average was wrong - meanwhile there was an awful lot of confusion - at about N a.m. on the over-the-counter trading desk at a major brokerage firm a veteran trader who buys and sells some of the most active stocks looked at a senior official and asked what 's going on - is the market up or down - at the time quotron was reporting that the industrial average was down N points - in fact it was up N - stark a vice president who heads the trading desk at dillon read capital corp. said that once she figured out the quotron numbers were wrong she called brokers to tell them - it 's been kind of to say the least she said - to matters further when ual corp. stock finally opened on the new york stock exchange at N a.m. the price was listed at $ N a share up about $ N from friday in fact its true price was $ N down $ N - that was the new york stock exchange 's - a spokesman cited a technical error and declined to elaborate - and there were other - when the market opened at N a.m. est a reporter for the reuters the industrial average 's drop as a N N decline when it really was down N N - it was a case of human error which we found almost immediately and corrected a spokesman for reuter in new york said - meanwhile some currency traders at west german banks in frankfurt said they sold dollars on the news and had to buy them back later at higher prices - but it was the quotron problems that had effects - dillon read 's ms. stark said in early afternoon that she was still prices and other data as subject to and she said portfolio managers continued to question the numbers they saw on the screen - it was the second time in less than a week that quotron has had problems the industrial average - at the start of trading last wednesday the average appeared to plunge more than N points - actually it was down only a few points at the time - quotron said that which lasted nine minutes resulted from a failure to adjust for a stock split at philip morris - a quotron spokeswoman said recent software changes may have contributed to yesterday 's problems - she said quotron switched to a backup system until the problems were corrected - today of all days she - the eyes of the world were watching us - steven f. was named a senior vice president of this graphics equipment company - he retains his current positions as chief strategic officer of am international and president of am ventures - houston attorney dale friend representing a plaintiff in a damage suit says he has negotiated a settlement that will strike a blow for his client - literally - it turns out mr. friend 's client parks of cincinnati did n't like the way defense attorney tom alexander acted during the legal proceedings - so she has agreed to monetary damages against mr. alexander 's client in return for the right to the attorney - ms. parks 's mother also gets to mr. alexander - so does mr. friend and his law partner - the bizarre arrangement grows out of mr. alexander 's representation of construction co. one of several defendants in a death lawsuit brought by ms. parks the widow of a construction worker killed in january N while working on a new houston convention center - last month mr. friend says mr. alexander 's associate agreed that would pay $ N as part of an overall settlement - but mr. alexander the deal at the last minute the plaintiff 's side - i never agreed to it mr. alexander says adding that it 's not necessary to pay these settlements - when ms. parks and her mother heard about what had happened mr. friend says they that they would like to give mr. alexander a good - mr. friend says he passed that along to his adversary and soon they were talking about the ground rules under which could keep its money and the plaintiffs could take a shot at mr. alexander - although time and place have yet to be determined some details are in place - mr. friend says he agreed to strike mr. alexander above the belt - ms. parks and her mother indicated they want to catch him from behind he says - mr. alexander for his part insisted that the ca n't their rights to anyone else ca n't use a blunt instrument and ca n't take a running start - mr. alexander says he the agreement which has n't been submitted to a judge as something of a joke - however he acknowledges they have the option of taking a at me if they really want to - mr. friend says his side is dead serious - although they do n't delivering any he says that mr. alexander will be asked to sign a release from liability just in case - after two years of drought it money in the stock-index futures markets yesterday - as financial markets rebounded trading volume in the chicago mercantile exchange 's huge standard & poor 's N stock-index futures pit soared reaching levels for the first time since october N - the sudden influx of liquidity enabled several traders to reap in a matter of minutes as prices soared traders said - guys were money in there today said john a futures broker for elders futures inc. in chicago - the s&p N futures contract which moves in of an index point under normal conditions jumped two to three points in seconds early yesterday after an initial downturn then moved strongly higher the rest of the day - each index point represents a $ N profit for each s&p N contract held - for the first time since the N crash traders said that they were able to trade several hundred s&p N contracts at a time in a highly liquid market - many institutions and individual investors have away from stock-index futures blaming them for speeding the stock market crash on black monday two years ago - since the crash many futures traders have n't assumed large positions for fear that the s&p N market with much of its customer order flow missing would dry up if prices turned against them - more than N traders the s&p N futures pit to await the opening bell - traders were shouting bids and offers a full five minutes before the start of trading at N am - the contract fell five points at the open to N the maximum opening move allowed under adopted by the merc to stem a market slide - but several traders quickly stepped up and bid for contracts driving prices sharply higher - the market near friday 's closing price of N for about a half hour moving several index points higher or lower in seconds then broke higher and did n't look back - the s&p N contract that expires in december closed up a record N points on volume of nearly N contracts - traders five feet from each other were making bids and offers that were a full point apart said one s&p N broker - you could buy at the bid and sell at the offer and make a fortune he - several of wall street 's largest securities firms including salomon brothers inc. and painewebber inc. were also large buyers traders said - salomon brothers was among the largest sellers of stock-index futures last week traders said - brokerage firms as a rule do n't comment on their market activity - unlike the week following black monday two years ago individual traders in the s&p N pit were also being about their one-day profits - with the fbi around here rights are a thing of the past said one trader referring to the federal investigation of futures trading that so far has resulted in N against individuals on the merc and the chicago board of trade - the market for $ N billion of high-yield junk bonds regained some of its footing as the dow jones industrial average rebounded from friday 's plunge - but the junk recovery led by the bellwether rjr holdings bonds was - no trading existed for the vast majority of junk bonds securities industry officials said - on friday trading in practically every issue ground to a halt as potential buyers fled and brokerage firms were unwilling to provide bid and offer prices for most issues - nothing traded on friday and people were n't really sure where the market should have opened yesterday said raymond of merchant banking at merrill lynch & co - but we had a fairly active day yesterday - at drexel burnham lambert inc. the leading underwriter of junk bonds i was prepared to be in a very bad mood tonight said david a junk bond trader - now i feel maybe there 's a little bit of euphoria - but before the stock market rebounded from a sharp early sell-off yesterday he said you could n't buy junk bonds and you could n't give them away - yesterday 's rally was led by rjr holdings N N N bonds which initially tumbled three points or $ N for each $ N face amount to N N before rebounding to N N - bonds issued by and american standard also showed big gains recovering almost all their losses from friday and early yesterday - but traders said the junk bond market increasingly is into a group in which trades can be executed easily and a larger group of bonds in which liquidity or the ability to trade without too much difficulty has steadily deteriorated this year - liquidity has n't returned to the vast middle ground of the market said mr. of merrill - the are still said mr. of drexel - analysts are concerned that much of the high-yield market will remain for investors - paul associate professor at the massachusetts institute of technology 's sloan school of management citing a pattern of junk-bond default rates that are low in the early years after issuance and rise later says we 're now in a period where we 're starting to see defaults from the big issue years of N to N - mark a senior vice president at standard & poor 's corp. confirms that there is increasing concern about the future liquidity of the junk bond market - junk bonds are a highly market said lewis vice chairman of smith barney harris upham & co - there 's a whole bunch of stuff that 's money good and a whole bunch of stuff that 's not so good - analysts at standard & poor 's say junk bond offerings by tightly stretched issuers seem to be growing - almost $ N billion of junk bonds that are considered include issues from sci tv gillette holdings not related to gillette co. furniture allied stores federated department stores national holdings leaseway transportation and price communications - you could still have some very bad times ahead said mr. - it 's possible to have a N N default rate in one year because we 're already seeing big problems in the midst of a pretty strong economy - i 'm certainly not comfortable saying we 've seen the bottom - but yesterday 's rally among good junk was a badly needed for the market - many issues bounced off the floor mr. said and benchmark junk issues recovered all of their losses from friday and early yesterday - in contrast he says the stock market gained back only about half what it lost friday and the government bond market lost about half what it gained friday - traders said yesterday 's rally was fueled by insurance companies looking for bargains after a drastic slide in prices the past month - in addition mutual funds did n't appear to be major sellers of high-yield securities as was expected - sometimes a is healthy said drexel 's mr. - people will learn to be more - if they do good credit analysis they will avoid the hand - i think the market is in good shape - should you really own stocks - that 's a question a lot of people are asking following the stock market 's stunning display of volatility - financially and by friday 's 190-point drop in the dow jones industrial average and yesterday 's rebound they 're wondering if an individual has any business being in the market - the answer say academic researchers money managers and investment specialists is yes as long as you approach the stock market as an investor - but they say people should n't try to be traders who buy and sell in an effort to ride the latest economic trend or catch the next hot stock - the case for owning stocks over the long-term is compelling - if you look at N years worth of investment history including the great depression and every bear market since stocks have outperformed almost everything an individual could have owned by a long shot says barry berlin vice president at first wachovia capital management - a dollar invested in the stock market in N would have grown to $ N by the end of last june according to laurence managing director at associates inc - but a dollar invested in long-term bonds in N would have grown to only $ N and a dollar put in treasury bills would equal a $ N - the longer the time period the less risk there is of losing money in the stock market - over time the odds increasingly favor the investor with a diversified portfolio - for instance ken gregory a san francisco money manager that if an investor holds a basket of stocks that tracks the standard & poor 's 500-stock index the chance of losing money is N N to N N over a 10-year period compared with N N over three years and N N over one year - if you do n't need the money for N years there 's a case for sticking to a steady core of stocks mr. gregory says - stock-market investments also help balance the other assets an individual owns says john jr. president of the institute of certified financial planners - stocks have a place in an investors ' portfolio along with real estate bonds international securities and cash he says - there are some important before investing in stocks individuals should have at least three to six months of living expenses set aside in the bank most investment advisers say - individuals also should focus on building equity in a home which provides some protection against inflation as well as a that can be in late in life to help cover the cost of retirement living - people also should n't invest money in stocks that they 'll need in the near future for example for college tuition payments or retirement expenses - you may have to sell your stocks at a time when the market takes a plunge says mr. a del calif. financial planner - but once the are covered then i would start to invest even if it 's as little as $ N says michael lipper president of lipper analytical services inc - he says individuals should consider not just stocks but other long-term investments such as high-quality bonds - despite the strong case for stocks however most pros warn that individuals should n't try to profit from short-term developments - it 's very difficult to do says donald holt a market strategist for morgan securities a los angeles brokerage firm - our markets move so fast and they are so volatile there 's no way the average investor can compete with the pros - individual investors face high transaction costs of moving in and out of the market - the cost of executing stock orders from brokerage to brokerage and with the size of the order but N N of the order 's value is an average says stephen boesel manager of t. rowe price 's growth and income mutual fund - and assuming their first investment is successful investors will have to pay taxes on their gains - that can reduce returns by a third or more once local taxes are included mr. lipper says - after that individual traders face the risk that the new investment they choose wo n't perform well so their trading costs could be sustained for nothing - it 's very tough for most individuals to the mutual funds or the market says mr. lipper - you should really think twice if you think you can the system - then too many individual investors lack the emotional makeup professionals say is needed to plunge in and out of the market - so what 's the best way to buy stocks - unless an individual has a minimum of between $ N and $ N to invest in stocks he 's still better off in mutual funds than in individual stocks in terms of getting enough attention from a competent broker says mr. lipper - still he adds i could see owning both given that individuals often have an advantage over big investors in special situations based on their own he adds - george douglas first vice president at drexel burnham lambert inc. says that individuals have a particular edge now in small to niche companies with exciting earnings prospects a traditional ground for small investors - this growth sector which usually carries a multiple about twice that of the standard & poor 's N happens to include some of the market 's most attractive bargains right now - it 's now selling at a multiple about even with the market says mr. douglas - moreover mr. douglas sees a revival of institutional interest in smaller growth stocks that could boost the performance of these stocks in the medium term - many big wall street brokerage firms who eliminated their research effort in stocks of emerging growth companies a few years ago are now coverage of this area he notes - we 're seeing a real turnaround in interest in small growth stocks he says - the pros advise individuals to stay away from the latest investment fad - they say that 's especially important this late in the growth phase of the economic cycle when there 's no robust bull market to bail investors out of their mistakes - friday 's correction presents a pretty good buying opportunity but let 's not speculate at this point in the business cycle says chief equity portfolio strategist at first boston corp - buy stocks on weakness for their long-term fundamentals he says - in the long run investment advisers say most investors will be better off using the averaging method of buying stocks - in this method a person invests a regular amount every month or quarter into the stock market whether the market is up or down - that cuts the risk mr. gregory the san francisco money manager points out - when the market is low you are buying more shares and when it 's high you 're buying fewer shares he says - otherwise if you put all your money in at one time by sheer bad luck you might pick a terrible time and have to wait three years to get even mr. gregory says - a disciplined program will work the best mr. boesel says - one of the hardest things to do is to buy stocks when the market is down he says - but that 's just the time when you should be buying them - compound annual returns including price changes and income from interest and dividends - \* actual performance not annualized - source associates inc - the following issues were recently filed with the securities and exchange commission co. initial public offering of two million shares of common stock of which N shares are being offered by the company and N shares by holders via blunt ellis & inc. and robert w. & co - giant industries inc. initial public offering of N common shares of which N will be sold by the company and the rest by holders via shearson lehman hutton inc. and inc - fund inc. initial offering of five million common shares via smith barney harris upham & co - overseas ltd. initial offering of four million common shares of which N million will be sold in the u.s. and the balance outside the u.s. via smith barney harris upham & co. and & co - donald trump who faced rising doubt about his bid for american airlines parent amr corp. even before a united airlines buy-out came apart friday withdrew his $ N billion offer - separately bankers representing the group trying to buy united 's parent ual corp. met with other banks about that purchase at a lower price possibly around $ N a share or $ N billion - but a lower bid could face rejection by the ual board - mr. trump who vowed wednesday to go forward with the bid said he was dropping it in light of the recent change in market conditions - he said he might now sell his amr stake buy more shares or make another offer at a lower price - the manhattan real-estate developer acted after the ual buyers failed to obtain financing for their earlier $ 300-a-share bid which sparked a selling panic among that into a 190-point drop friday in the dow jones industrial average - news about ual and amr whose shares never reopened after trading was halted friday for the ual announcement sent both stocks in composite trading on the new york stock exchange - ual tumbled $ N to $ N on volume of N million shares and amr declined by $ N to $ N as N million shares changed hands - together the two stocks havoc among takeover stock traders and caused a N N drop in the dow jones transportation average second in size only to the stock-market crash of oct. N N - some said friday 's market debacle had given mr. trump an excuse to bail out of an offer that showed signs of even before problems emerged with the ual deal - after reaching an intraday high of $ N the day mr. trump disclosed his bid oct. N amr 's stock had retreated as low as $ N last week - some takeover stock traders had been betting against mr. trump because he has a record of disclosing stakes in companies that are potential takeover targets then selling at a profit without making a bid - he still has n't proven his as a artist said airline analyst kevin murphy of morgan stanley & co - he 's done this thing where he 'll buy a little bit of a company and then trade out of it - he 's written this book the art of the deal - why does n't he just follow through on one of these things - mr. trump withdrew his bid before the amr board which is due to meet tomorrow ever formally considered it - amr had weighed a wide range of possible responses from flat rejection to and leveraged buy-outs that might have included either employees a buyer such as texas billionaire robert bass or both - amr had also sought to mr. trump in congress by lobbying for legislation that would have bolstered the authority of the transportation department to reject airline buy-outs - yesterday mr. trump tried to put the blame for the collapse of the ual deal on congress saying it was rushing through a bill to protect amr executives - i believe that the perception that legislation in this area may be hastily approved contributed to the collapse of the ual transaction and the resulting disruption in the financial markets experienced this past friday mr. trump wrote members of congress - amr declined to comment and mr. trump did n't respond to requests for interviews - mr. trump never said how much amr stock he had bought only that his holdings were substantial - however he only received federal clearance to buy more than $ N million of the stock on sept. N when the price rose $ N a share to $ N - between then and his bid on oct. N the price between $ N and $ N - in an attempt to persuade investors that his bid was n't just a stock play mr. trump promised last week to notify the market before selling any shares - amr was trading at around $ N yesterday before his withdrawal announcement then immediately fell to about $ N - assuming that he paid a rough average price of $ N a share and assuming he did n't sell before his announcement reached the market mr. trump could be sitting with a modest loss with the stock at $ N - some analysts said amr chairman robert crandall might seize the opportunity presented by the stock price drop to protect the nation 's largest airline with a defensive transaction such as the sale of stock to a friendly holder or company employees - however other knowledgeable observers said they believed mr. crandall and the amr board might well decide to tough it out without taking any extra steps - some analysts said they believed mr. trump whose had been viewed by some as a reason to believe he would n't back out might come back with a lower bid - ray of dillon read & co. said mr. trump is stepping back and waiting for the dust to settle - i 'm sure he still wants amr - but others remained skeptical - i was never sure donald trump really wanted to take amr said john a bond analyst with shearson lehman hutton inc - what happened with united was a way for him to out - mr. trump never obtained financing for his bid - that skepticism would leave him with an even greater credibility problem should he return that would him in any effort to oust the board in a proxy fight - meanwhile citicorp and chase manhattan corp. the two lead lenders on the ual buy-out met with other banks yesterday to determine if they would be willing to finance the buy-out at a lower price - officials familiar with the talks said citicorp had discussed lowering the offer to $ N a share but said that price was a talking point and that no decision has been made - at $ N a share the group would have to borrow about $ N billion from banks - the first ual deal unraveled after citibank and chase could n't raise $ N billion - citibank and chase had agreed to commit $ N billion and said they were highly confident of raising another $ N billion - together citicorp and chase received $ N million in fees to raise the rest of the financing - but other banks balked at the low interest rate and banking fees the ual group was willing to pay them - officials familiar with the bank talks said the ual buy-out group ual pilots management and british airways plc is now willing to pay higher bank fees and interest but is n't likely to boost its $ N million equity contribution - nor is the group likely to come forward with a revised offer within the next N hours despite the hopes of many traders - the group 's advisers want to make certain they have firm bank commitments the second time around - even if the buy-out group is able to obtain financing the transaction still faces obstacles - ual 's board could reject the new price as too low especially since there are n't any competing bids - los angeles investor marvin davis whose $ offer was rejected by ual 's board has n't shown signs of pursuing a $ 300-a-share bid he made last month - in addition the coalition of labor and management longtime enemies who joined forces only under the threat of mr. davis 's bid could break apart now - the group 's resilience gets its first test today when N top pilot union leaders outside chicago in a previously scheduled meeting - union chairman rick faces the tough task of explaining why banks refused to finance a buy-out the members approved last week - the pilot union is to pursue an acquisition whatever the board decides - but if the board a reduced bid and decides to explore other alternatives it could transform what has been a process into an one - the pilots could play by noting they are crucial to any sale or restructuring because they can refuse to fly the airplanes - if they were to insist on a low bid of say $ N a share the board might n't be able to obtain a higher offer from other bidders because banks might hesitate to finance a transaction the pilots oppose - also because ual chairman stephen wolf and other ual executives have joined the pilots ' bid the board might be forced to exclude him from its deliberations in order to be fair to other bidders - that could cost him the chance to influence the outcome and perhaps join the winning bidder - influential members of the house ways and means committee introduced legislation that would restrict how the new savings-and-loan bailout agency can raise capital creating another potential obstacle to the government 's sale of sick thrifts - the bill whose backers include chairman dan d. ill. would prevent the resolution trust corp. from raising temporary working capital by having an bank or thrift issue debt that would n't be counted on the federal budget - the bill intends to restrict the rtc to treasury borrowings only unless the agency receives specific congressional authorization - such agency borrowing is unauthorized and expensive far more expensive than direct treasury borrowing said rep. stark d. calif. the bill 's chief sponsor - the complex financing plan in the s&l bailout law includes raising $ N billion from debt issued by the newly created rtc - this financing system was created in the new law in order to keep the bailout spending from swelling the budget deficit - another $ N billion would be raised through treasury bonds which pay lower interest rates - but the rtc also requires working capital to maintain the bad assets of thrifts that are sold until the assets can be sold separately - that debt would be paid off as the assets are sold leaving the total spending for the bailout at $ N billion or $ N billion including interest over N years - it 's a problem that clearly has to be resolved said david executive director of the rtc - the agency has already spent roughly $ N billion selling N insolvent s&ls and it is likely to sell or merge N by the time the bailout concludes - other working capital he said the rtc would be forced to delay other thrift resolutions until cash could be raised by selling the bad assets - we would have to wait until we have collected on those assets before we can move forward he said - the complicated language in the huge new law has the fight - the law does allow the rtc to borrow from the treasury up to $ N billion at any time - moreover it says the rtc 's total obligations may not exceed $ N billion but that figure is derived after including notes and other debt and from it the market value of the assets the rtc holds - but congress did n't anticipate or intend more public debt say opponents of the rtc 's plan and rep. charles d. n.y said the rtc oversight board has been in not keeping congress informed - that leads to a proposal like the one from ways and means which seems to me sort of he said - the rtc is going to have to pay a price of prior on the hill if they want that kind of flexibility - the ways and means committee will hold a hearing on the bill next tuesday - we 're about to see if advertising works - hard on the heels of friday 's 190-point stock-market plunge and the uncertainty that 's followed a few big brokerage firms are rolling out new ads a familiar message keep on investing the market 's just fine - their mission is to keep clients from the market as individual investors did in after the crash in october - just days after the N crash major brokerage firms rushed out ads to calm investors - this time around they 're moving even faster - painewebber inc. a new television commercial at N p.m. edt yesterday and had it on the air by last night - fidelity investments placed new ads in newspapers yesterday and wrote another new ad appearing today - shearson lehman hutton inc. by yesterday afternoon had already written new tv ads - it considered running them during tomorrow night 's world series broadcast but decided not to when the market recovered yesterday - other brokerage firms including merrill lynch & co. were out potential new ad strategies - the brokerage firms learned a lesson the last time around when frightened investors flooded the phone lines and fled the market in a panic - this time the firms were ready - fidelity for example prepared ads several months ago in case of a market plunge - when the market went into its free fall friday afternoon the investment firm ordered full pages in the monday editions of half a dozen newspapers - the ads touted fidelity 's automated beneath the huge headline fidelity is ready for your call - a fidelity spokesman says the which already was operating but which many clients did n't know about received about double the usual volume of calls over the weekend - a lot of investor confidence comes from the fact that they can speak to us he says - to maintain that dialogue is absolutely crucial - it would have been too late to think about on friday - we had to think about it ahead of time - today 's fidelity ad goes a step further encouraging investors to stay in the market or even to plunge in with fidelity - the headline diversification it based on the events of the past week all investors need to know their portfolios are balanced to help protect them against the market 's volatility - it goes on to plug a few diversified fidelity funds by name - painewebber also was able to gear up quickly thanks to the N crash - in the aftermath of the N debacle the brokerage firm began taping commercials in-house ultimately getting its timing down fast enough to tape a commercial after the market closed and rush it on the air that night - it also negotiated an arrangement with cable news network under which would agree to air its last-minute - the new painewebber commercial created with ad agency saatchi & saatchi co. features mary farrell one of the firm 's most visible investment strategists particularly bullish - taped just as the market closed yesterday it offers ms. farrell advising we view the market here as going through a relatively normal cycle - we continue to feel that the stock market is still the place to be for long-term appreciation - the spot was scheduled to appear three times on last night - painewebber considered an even harder sell recommending specific stocks - instead it settled on just urging the clients who are its to keep that money in the market - we 're saying the worst thing that anyone can do is to see the market go down and dump everything which just drives the prices down further says john painewebber 's director of advertising - if you owned it and liked it friday the true value has n't changed - he adds this is n't N - with the market and then closing up more than N points yesterday investment firms had to constantly revise their approach - at shearson lehman executives created potential new commercials friday night and throughout the weekend then had to yesterday afternoon - the plan had been to make one of shearson 's black-and-white where we stand commercials which have been running occasionally in response to news events since N - the ad would have run during the world series tomorrow replacing the debut commercial of shearson 's new ad campaign leadership by example - but in a meeting after the market closed yesterday shearson executives decided not to go ahead with the stock-market ad - we do n't think at this point anything needs to be said - the market seems to be out we 're taking a attitude says b. stewart executive vice president of marketing - in any case the brokerage firms are clearly moving faster to create new ads than they did in the fall of N - but it remains to be seen whether their ads will be any more effective - in N despite a of ads from most of the major investment firms individuals ran from the market en - now the firms must try their hardest to prove that advertising can work this time around - ad notes - arnold advertising - edward former chairman of della femina mcnamee reached an agreement in principle to acquire a majority stake in arnold advertising a small boston shop - terms were n't disclosed - mr. who resigned his della femina post in september becomes chairman and chief executive of arnold - john the agency 's president and chief executive will retain the title of president - separately mcdonald 's corp. oak ill. named arnold to handle its estimated $ N million cooperative ad account for the hartford conn. area - that account had been handled by della femina mcnamee wcrs - education ads - a ad supplement to business week 's special corporate elite issue calls on business leaders to use their clout to help solve the nation 's education crisis - the supplement the largest ever for the magazine includes ads from N corporate advertisers and off a two-year business week initiative on education - the magazine will distribute N N of the gross revenues from the supplement as grants to innovative teachers - you know what the law of averages is do n't you - it 's what N explains why we are like well ourselves rather than jackson N that it 's possible to in a lake that averages two feet deep and N predicts that N placed before N would produce N rock roll - baseball that game of the long haul is the sport of the mean and the mean law caught up with the san francisco giants in the world series last weekend - the team that dumped runs by the bushel on the chicago cubs in the national league playoffs was held to just one in two games by the oakland a 's the gang that had been done similarly by the los angeles and in last year 's - much of the damage was accomplished by a 's who had some catching up to do - in game two on a cool sunday evening in this land of perpetual autumn a lot of the catching up was done by the a 's terry - he hit a N pitch from rick into the stands in inning four to stretch his team 's lead from N to a decisive N where it stayed - so what if had struck just seven home runs in N regular-season games and in the seventh position of the a 's lineup - if you get your pitch and take a good swing anything can happen he later - on saturday night quite a few of the boys in green and gold away successes to the pain of past and no doubt future - mark the big oakland first had three hits in four at two more than he 'd had in the series in which he 'd gone - the N through N the bottom of the order got seven of their team 's N hits and scored four of its runs in a N decision - dave stewart held the giants to five hits to account for the zero on the other side of the saturday - that he was the a 's during its american league campaign with a N mark plus two wins over toronto in the playoffs indicates he may have some evening up coming but with the way his is that might not be this week - the same goes for mike moore another veteran who early struggles to permit the giants but a run and four hits in seven in sunday 's contest - every guy they put out there had a better than the guy before giant manager roger craig - he 's an who 's one of the leading of the fashionable delivery which looks like a until it beneath the bat - the of the is that the a 's go into san francisco 's candlestick park tonight up two games to none in the - the to with here says that about three of four clubs N of N that took N series leads went on to win it all - that 's not an average to giant - one might think that the home fans in this series of the subway called bart that 's a better name for a public than desire do n't you think would have been over the proceedings but they them in relative calm - of the two sat side by side in the seats of oakland and while they cheered their and the opposition advanced no further at least as far as i could see - a few folks even showed up wearing bearing the colors and of both teams - i 'm for the giants today but only because they lost yesterday - i love both - the only thing i 'm for is for the series to go seven games said david williams a sacramento at the before sunday 's go - the above represents a of either or - i choose to believe it 's the latter although it probably springs from the fact that just about everyone out here including the a 's and giants is originally from somewhere else - it to say that if this were a new york series or one between the chicago cubs and white it 's possible you 'd need police in every other seat to separate opposing fans and only the would their - anyway the a 's gave you a lot of heroes to root for - in the opening game besides and stewart there was walt weiss a who had lost a couple months of the season to surgery - he was in game two moved a along in the a 's second inning and for his team 's final tally - such is his reputation among the east bay that when he hit his first career home run last season the fan who caught it agreed to turn the ball over to him in return for an - not his 's - an a 's of the second game was henderson who the hot side of the equation - he toronto in the playoffs with six hits seven walks and eight stolen bases in N at and continued that by going at the plate sunday along with walking stealing a base and scoring a run - when you 're in the you see every ball he - the cold guys in the set were will clark kevin mitchell and williams the giants ' N - they combined for N hits six home runs and N runs in in the five games against the cubs - they went a collective here with zero and - it 's that last set of numbers as much as anything else that gives the giants hope in the series games to come - i believe in the law of averages declared san francisco coach dusty baker after game two - i 'd rather see a who 's hot come up for the other side than a good who 's cold - but the old offered no prediction about when good times would return to his side - when it goes you never know when you 'll get it back he said - that 's baseball - ncr corp. reported a N N drop in third-quarter net income citing intense competition that caused its gross profit margins to dip - net income for the quarter fell to $ N million from $ N million roughly what analysts had expected - but per-share profit dropped only N N to $ N a share from $ N a share as the company continued its stock buy-back plan - average shares outstanding dropped to N million from N million - revenue fell N N to $ N billion from $ N billion - the computer maker which sells more than half its goods outside the u.s. also said the negative effect of a stronger u.s. dollar will affect its fourth-quarter performance and make it difficult to better N results - ncr said revenue declined both in the u.s. and overseas reflecting a world-wide softening of the computer markets - the company however said orders in the u.s. showed good gains during the latest quarter - analysts estimate those gains at N N to N N a good part of it coming from large orders placed by a few of ncr 's major customers - in addition to a general slowing of the computer industry ncr which sells automated teller machines and computerized cash is also affected by the retail and financial sectors areas of the economy that have generally not been robust notes g. an analyst for salomon brothers inc - these factors combined with a strong dollar should affect the current quarter 's results ncr said - in the year-earlier fourth quarter ncr had profit of $ N million or $ N a share on revenue of $ N billion - mr. said he lowered his full-year estimates for N to $ N a share from $ N a share - revenue projections were slashed to $ N billion from $ N billion - last year ncr had net income of $ N million or $ N a share on $ N billion in revenue - for the nine months the company 's earnings fell N N to $ N million or $ N a share from $ N million or $ N a share - revenues declined N N to $ N billion from $ N billion - in new york stock exchange composite trading yesterday ncr shares fell N cents to close at $ N - concerning your sept. N article wall street firms link analysts ' pay to performance i 'm that wall street is finally in to the hard cold facts of the real working world - if the firms are serious however why limit the practice to the poor analysts whose ability to see into the future is fragile at best - why not extend the same harsh standards to the sales force and pay brokers a base salary with annual bonus based on how much money they made for their clients during the year - that should stop a lot of and produce a stock market driven only by professional concern careful thought and good sense - now would n't that be a - newport news va - steve clark a shearson lehman hutton inc. trader reported for work at N a.m. two and a half hours before the usual monday morning strategy meeting - at jefferies & co. j. francis did n't reach the office until N a.m. but then he had been up most of the night at home - i had calls all night long from the states he said - i was up every hour N N N N - people are looking for possible opportunities to buy but nobody wants to stick their out - for many of london 's securities traders it was a day that started nervously in the small hours - by the selling was at fever - but as the day ended in a wall rally the city a sigh of relief - so it went yesterday in the trading rooms of london 's financial district - in the wake of wall street 's plunge last friday the london market was considered especially vulnerable - and before the opening of trading here yesterday all eyes were on early trading in tokyo for a clue as to how widespread the fallout might be - by the time trading officially got under way at N a.m. the news from asia was in - and it left mixed signals for london - tokyo stocks closed off a significant but N N on thin volume hong kong stocks declined N N in orderly trading - at jefferies ' trading room on circus a circle at the edge of the financial district desktop computer screens displayed the london market 's major barometer the financial times-stock exchange N share index - red figures on the screens indicated falling stocks blue figures rising stocks - right away the outnumbered the blues N to N as the index opened at N off N points or N N - i see concern but i do n't see any panic said mr. a big new york native who runs the office - the jefferies office a branch of the los angeles-based firm played it seeking to avoid risk - this is not the sort of market to have a big position in said david smith who heads trading in all non-u.s. stocks - we tend to run a very tight book - jefferies spent most of its in the morning trying to match buyers and sellers and there were n't many buyers - all the takeover stocks scottish & b.a.t are getting pretty well this morning mr. smith said - seconds later a sell order for scottish & came in - for the third time in N minutes a trader next to mr. smith left the area to have a cigarette - on the screens only two blue figures remained but the index had recovered a few points and was off about N - because tokyo did n't collapse let 's pick up a little stock mr. smith said - he targeted N shares of reuters and a to call up on his screen other dealers ' price quotes - the vivid yellow figures showed the best price at N pence $ N and mr. smith 's traders started putting out - but the market a serious buyer on a day dominated by selling and the quotes immediately jumped to N pence - when i want to buy they run from you they keep changing their prices mr. smith said - it 's very frustrating - he temporarily abandoned his search for the reuters shares - by this time it was N a.m. in new york and mr. smith a call from a new york customer wanting an opinion on the british stock market which had been having troubles of its own even before friday 's new york market break - fundamentally dangerous mr. smith said almost in a fundamentally weak fairly vulnerable still extremely poised - we 're in for a lot of turbulence - he was right - by midday the london market was in full retreat - it 's falling like a stone said danny a pit trader who was standing outside the london international financial futures exchange - only half the usual crowd gathered at the tony & wine bar on old broad street nearby - conversation was subdued as most watched the latest market statistics on television - at N p.m. the index hit its low N off N points - france opened the limit down off at least N N if you could calculate the index which you could n't mr. clark the shearson trader said early in the afternoon - spain is down N N and suspended sweden 's down N N norway N N - this market has been very badly damaged - as N p.m. wall street 's opening time shearson traders and salesmen traded bets on how low the new york market would open - in the center of the trading floor chief trader roger and two colleagues scrambled for the telephones as soon as the new york market opened more than N points in the first few minutes - they saw an opportunity created by the sell-off - as wall street traders dumped american depositary receipts in jaguar plc mr. and trader sam bought them to in the - investors here still expect ford motor co. or general motors corp. to bid for jaguar - suddenly after about N minutes the u.s. markets rallied - the mmi has gone better shouted one trader at about N london time as the u.s. major markets index contract suddenly indicated a - as wall street strengthened the london trading room went wild - traders shouted as their screens posted an loss on wall street - then nine minutes later wall street suddenly rebounded to a gain on the day - rally rally rally shouted shearson trader andy rosen selling more jaguar shares - this is panic buying - as the london market rallied some whether the weekend of worrying and jitters had been worth it - the london index closed at N its high for the day off N or about N N - ambassador paul 's statement notable & sept. N if you have a million people working for you every bad thing that has one chance in a million of going wrong will go wrong at least once a year is a pretty negative way of looking at things - is n't it just as fair to say that if you have a million people working for you every good thing that has one chance in a million of going right will go right at least once a year - do n't be such a mr. ambassador - frank - the house aviation subcommittee approved a bill that would give the transportation secretary authority to review and approve leveraged buy-outs of major u.s. airlines - the collapsed plan to acquire ual corp. parent of united airlines spurred quick action on the legislation introduced wednesday and approved by the subcommittee on a voice vote yesterday - the bill is expected to be taken up by the public works and transportation committee tomorrow and a floor vote by next week will be urged - the measure drew criticism from the bush administration and a shot from financier donald trump who yesterday withdrew his takeover bid for amr corp. the parent of american airlines - in a letter to subcommittee chairman james d. minn. mr. trump criticized the bill as an explicit effort to thwart his bid for amr and said it contributed to the collapse of the deal - deputy transportation secretary also sent a letter to express the administration 's opposition to the bill in its present form - rep. brushed off mr. trump 's allegations as an excuse for his own deal failing - he also said the fact that the other letter had n't come from transportation secretary samuel skinner indicated there is room in the administration 's position - mr. and other committee members repeatedly stressed that the legislation was n't a response to any particular market situation - but they cited the ual and amr examples as reasons to move quickly to enact this legislation - aides both in the house and senate said the withdrawal of the trump bid for amr is n't likely to efforts to push the legislation - it 's still on the fast track and we still want to do it said one senate aide - the bill is aimed at addressing the concern that an airline might sacrifice costly safety measures to pay off the debt incurred in a leveraged buy-out - currently the transportation secretary does n't have clearly established authority to block mergers but can take the drastic step of the operating certificate of any carrier the official considers - supporters of the legislation view the bill as an effort to add stability and to the process and to preserve the safety and fitness of the industry - in general the bill would give the transportation department a 30-day review period before N N or more of the voting stock of a major u.s. air carrier could be acquired - it also would require the acquiring party to notify the transportation secretary and to provide all information relevant to determining the intent of the acquisition - the bill would allow the secretary to reject a buy-out if sufficient information has n't been provided or if the buy-out is likely to weaken the carrier financially result in a substantial reduction in size of the airline through disposal of assets or give control to a foreign interest - if more information is needed the secretary would have authority to extend the review period N days - all the witnesses both congressmen and industry experts expressed support for the bill in order to prevent from in on airline profits at the expense of safe service - but several committee members some backing mr. trump 's claim that the threat of regulation caused the failure of the ual deal and the stock-market plunge - one of the major concerns expressed by the was that large airlines would be prohibited from themselves of smaller entities and producing independent companies - in a possible prelude to the of talks between boeing co. and striking machinists union members a federal mediator said representatives of the two sides will meet with him tomorrow - it could be a long meeting or it could be a short one said doug hammond the mediator who called the agreement to meet a first step toward a of negotiations - we 're encouraged that talks are scheduled again but beyond that we have made no expression of expectations a boeing spokesman said - the machinists union has rejected a three-year contract offer that would have provided a N N wage increase over the life of the pact plus some bonuses - currently average pay for machinists is $ N an hour boeing said - now in its 13th day the strike has about N machinists and has started to delay delivery of some - with a strike fund of about $ N million the union had said it was prepared for a long strike - after the third week on strike union members will begin receiving $ N a week from the fund - work at boeing continues with supervisors and other personnel the lines - and at the company 's wichita kan. plant about N of the N machinists still are working boeing said - under kansas laws contracts can not require workers to be union members - boeing has declined to say how many employees are working at its giant wash. plant - union officials could n't be reached for comment - dpc acquisition partners a hostile suitor for dataproducts corp. said it intends to launch a tender offer for the computer printer maker 's common stock - dpc a group led by the new york investment firm inc. also said it plans to file preliminary materials with the securities and exchange commission regarding a shareholder solicitation to oust dataproducts ' board - dpc holds a N N stake in dataproducts and made a $ bid for the company in may but dataproducts management considered the $ N million proposal - a dpc spokesman declined to elaborate on the group 's new plan - in american stock exchange composite trading yesterday dataproducts shares jumped N cents to close at $ N - dataproducts which had been seeking a buyer for several months announced a restructuring plan in september and took itself off the auction block - the company 's restructuring includes plans to split into three sectors to phase out domestic printer manufacturing operations and to sell its new england subsidiary - as part of the plan dataproducts announced a pact to sell $ N million of its real estate holdings to properties inc. a unit of canada 's corp - jack davis dataproducts ' president chairman and chief executive officer said the company is at a loss to understand dpc 's intentions - he called today 's announcement and and said the company intends to proceed with its restructuring - share prices plummeted across europe yesterday in response to friday 's new york sell-off but some issues staged a late comeback after wall street opened without another rout - european investors have further reason for optimism today after the u.s. rebound - the frankfurt stock exchange which closed before the new york exchanges opened was the hardest hit of the major european markets with the dax index dropping N N - in london prices plummeted in early trading and were off as much as N N before coming back strong after the new york opening to close down only N N - west german economics minister helmut said in my view the stock market will stabilize relatively quickly - there may be one or other psychological or technical reactions but they are n't based on fundamentals - the economy of west germany and the ec european community is highly stable - paris which has been the center of speculation fever in recent weeks also was hard hit - share prices fell in milan amsterdam zurich madrid and stockholm - prices in brussels where a computer breakdown disrupted trading also tumbled - following is a breakdown of major market activity - frankfurt - one of the sharpest declines came in the financial center of europe 's strongest economy - the dax index of N west german blue chips plunged N N a one-day record out the summer 's gains - the index closed at N down N points - by comparison two years ago on black monday the new index would have dropped N N according to a projection by the exchange - investors may have reacted so strongly to friday 's u.s. stock market loss because they had vivid memories of the frankfurt exchange 's losing N N of its value in the N crash and its wake - this time however many small investors may have been hurt by acting so swiftly - they all went in the wrong direction said andreas an investment adviser for the bank in 's frankfurt branch - he said he told clients to buy selected west german blue chips after they fell by about N N - after the opening was delayed N minutes because of the crush of sell orders frankfurt 's normal trading session was extended N minutes to handle the heavy volume - the beginning was chaotic said nigel a broker for commerzbank ag - it took of an hour before enough prices could be worked out to get a reading on the market - institutional investors and bankers many of whom spent the night before in their offices watching far eastern markets were cautiously optimistic after the mild N N decline in tokyo stock prices - everybody was still confident including most institutional investors - that is why everybody was a little surprised by the storm of sell orders from small private investors said a senior trader for - some big institutions including banks began picking up shares late yesterday but most investors wanted to see what would happen in new york before acting - but even if wall street continues to stabilize analysts here say the latest blow to investor confidence could inhibit a swift recovery for the frankfurt exchange which already was showing signs of weakness after the dax had slipped from a N high of N on sept. N - some of west germany 's chips took some of the biggest hits - a N N drop for ag and dresdner bank ag 's N N decline were especially for their respective boards whose plans for major rights issues in november could now be in jeopardy - dresdner bank last month said it hoped to raise N billion marks $ N million by issuing four million shares at N marks each - yet yesterday 's market dresdner 's share price by N marks to N marks a share leaving little incentive for investors to subscribe to the standing price unless the market quickly - london - headed toward a record drop at midday the london stock market two-thirds of its losses in the wake of new york 's early rally - the financial times-stock exchange N share index closed off N points at N its high for the day after having plunged N points at N p.m - it was big institutions such as union insurance group scottish amicable investment managers and standard life assurance co. that the rally - attracted by low prices and encouraged by new york 's performance they up equities across the board - volume was N million shares more than triple recent levels - paris - late buying gave the paris a after its free fall early in the day - the general index ended down N N at N a drop of N points from friday - there was a volatility in the market that i have never seen before said a partner in brokerage firm - when wall street turned around shortly after the opening there was panic buying in paris - brokers said that as the news spread that wall street was moving up traders who had called to place sell orders changed their line in ordering buys instead - trading was driven primarily by small investors and speculators with large institutions waiting on the sidelines until late in the day - when wall street turned however the big boys entered the market looking for bargains - j.p. morgan & co. swung to a loss in the third quarter while ncnb corp. reported net income more than doubled and security pacific corp. net rose N N - j.p. morgan & co - j.p. morgan as expected posted a $ N billion net loss for the quarter reflecting the new york bank 's decision last month to add $ N billion to reserves for losses on loans to less-developed countries - the reserve addition placed the parent of morgan guaranty trust co. among a few major u.s. banks that have covered nearly all their medium and long-term portfolios to less-developed countries with reserves - the latest quarter 's loss $ N a share - in the year-earlier quarter morgan earned $ N million or $ N a share - george m. analyst at prudential-bache securities inc. called the results mildly disappointing - excluding the $ N billion provision and allowing for the taxes morgan paid earnings were about N cents a share mr. said - in new york stock exchange composite trading yesterday morgan climbed $ N a share to $ N - net interest income sank N N in the quarter to $ N million from $ N million - the interest rate on short-term funds which banks borrow to finance longer-term loans to customers was sharply higher morgan said - morgan received $ N million of interest payments on its medium and long-term brazilian loans had they been interest net interest income would have been $ N million higher in the quarter morgan said - such loans to argentina also remain classified as costing the bank $ N million of interest income in the third period - income from sources other than interest climbed N N to $ N million reflecting higher and other fees and gains on sales of investment securities - these increases were partly offset by lower income the bank said - non-interest expenses grew N N to $ N million - ncnb corp - ncnb corp. 's net income more than doubled in the period largely because of continued strong performance by the bank 's texas operations - the charlotte n.c. company said earnings rose to $ N million or $ N a share from $ N million or N cents a share a year earlier - the latest quarter included a gain of $ N million or N cents a share related to the purchase of the remaining N N of ncnb texas national bank from the federal deposit insurance corp - the strong performance however with an unexpectedly large increase in the size of ncnb 's problem loans particularly in the southeast - in the third quarter nonperforming assets jumped to $ N million or N N of net loans and leases from $ N million or N N in the second quarter - totaled $ N million or N N in the year-ago third quarter - included in the increase in the most recent quarter is a $ N million loan which ncnb said it expects to be fully repaid with no loss early in the fourth quarter - the deterioration in credit quality offset strong loan growth of N N in ncnb 's southeast operations as well as a N N growth in deposits resulting from an aggressive marketing campaign - the higher rates paid on deposits also helped squeeze ncnb 's net interest margin in the southeast to N N from N N a year earlier - in big board composite trading yesterday ncnb jumped $ N a share to $ N - results were released after the market closed - ncnb texas national formed from the of of the failed first corp. of dallas contributed $ N million to ncnb 's bottom line in the third quarter - ncnb said its third-quarter results reflect N N of earnings of the texas operation since aug. N - ncnb raised some $ N billion in new capital during the quarter to complete the ncnb texas purchase and to acquire several small failed thrifts to fill out its regional franchise - last week the banking company said it purchased both freedom savings & loan association tampa fla. and university federal savings association of san antonio texas for $ N million - in the first nine months ncnb 's net income climbed N N to $ N million or $ N a share from $ N million or $ N a share a year earlier - security pacific corp - security pacific 's earnings growth slowed in the third quarter but the los angeles bank holding company was still able to post a N N increase in net income because of robust growth in residential real-estate and consumer loans - net rose to $ N million or $ N a share from $ N million or $ N a share a year earlier - the company said the gain resulted mainly from a $ N million increase in net interest income reflecting a N N increase in real estate loans mainly residential and a N N rise in consumer loans - these loans in effect replaced some assets such as loans which were allowed to decrease - as a result security pacific 's net interest margin fell only N basis points a more mild decrease than some major banks outside california which have been reporting more sluggish earnings - security pacific shares closed at $ N down N cents in big board composite trading - the earnings represent a N N return on assets for security pacific and an N N return on equity - the loan growth offset continuing real-estate loan losses in the depressed arizona market - security pacific reported a N N increase in net credit losses for the quarter to $ N million from $ N million in the year-ago period - nonperforming loans grew slightly to $ N billion at sept. N from $ N billion a year ago - security pacific 's loan-loss provision was down N N or $ N million because it added to its reserve the year before - non-interest income fell N N in the quarter mainly because of an unusual gain a year earlier from the sale of hong kong banking operations - non-interest expense grew only N N in the period - for the nine months net rose N N to $ N million or $ N a share from $ N million or $ N a share a year earlier - lin broadcasting corp. said it wo n't take a position on a revised tender offer by mccaw cellular communications inc. to buy lin and has asked for of the offer - the new offer which seeks N N of the cellular and broadcasting concern is for $ N a share for N million lin shares - mccaw 's revised tender offer would require mccaw to begin an auction process in july N that would buy out remaining holders at a per-share price roughly equivalent to what a third party might then have to pay for all of lin - lin is asking mccaw to clarify its tender offer which challenges an agreement between bellsouth corp. and lin to merge their businesses - bellsouth has notified lin that it would shortly respond to the mccaw proposal in as full and effective a manner as is - the lin board said holders may be by the provision in the mccaw proposal that guarantees private market value after five years for the remaining shares - mccaw has no obligation to purchase and the definition of private market value is uncertain the lin board said - the board added that mccaw would be able to control lin 's operations and could therefore operate lin in a manner which could its private market value and to a in five years - in national over-the-counter trading lin closed at $ N down $ N - a group of institutional investors in telerate inc. said that dow jones & co. 's $ offer for the electronic financial information services company is grossly inadequate - in a letter filed with the securities and exchange commission the group which holds about N million telerate shares or about N N of the shares outstanding said at present none of us believes an offer for less than $ N per share would be fair and some believe that $ N is too low - the letter was dated oct. N - in composite trading on the new york stock exchange telerate shares closed yesterday at $ N down N cents a share - dow jones publisher of the wall street journal has launched an $ or $ N million tender offer to acquire the remaining telerate shares outstanding dow jones owns N N of telerate - telerate has rejected the offer which expires nov. N - the group includes cos. and various affiliates based in boston wells fargo bank san francisco the california public employees retirement system sacramento calif. and t. rowe price associates inc. baltimore - among other issues the group 's letter said it has concerns as to whether dow jones 's offer meets the applicable requirements of procedural fairness - a spokesman for dow jones said he had n't seen the group 's filing but added obviously dow jones with their conclusions - our offer is to buy any and all shares tendered at $ N a share - u.s. trade representative carla hills said the first panel set up under the free trade agreement has ruled that canada 's restrictions on exports of pacific and violate the accord - mrs. hills said the u.s. and canada have until nov. N to resolve the dispute - if a solution is n't reached by then she said the u.s. would have the right to suspend some trade concessions to canada equivalent in value to the losses suffered by u.s. companies in alaska and the pacific northwest - however in canadian trade minister john said the panel accepted the legitimacy of canada 's position on the use of these landing requirements to and manage these important - questioned about the in the u.s. and canadian government views of the panel 's report an aide for mrs. hills said the panel had clearly ruled that the canadian trade restrictions are illegal - the u.s. trade representative declined to put a dollar estimate on the losses resulting from the canadian export restrictions - canada initially had an export prohibition that was replaced by regulations requiring that such fish had to be brought in british columbia by commercial prior to export - this action was defended by the canadian government on conservation grounds - mrs. hills said yesterday that the panel rejected this canadian government argument - we fully expect that canada will comply with the panel 's ruling that the landing requirement also must be ended she said - earlier an international panel set up under the rules of the general agreement on tariffs and trade in geneva determined that the original canadian restrictions violated gatt rules - mrs. hills said the u.s. wo n't accept any delays after nov. N because u.s. firms enter into contracts in the fall to purchase the next season 's catch - she said the canadian restrictions must be removed before such contracts are concluded - idle thought - to spend a idle day when duty calls to pay no to while the precious hours away character is what you need - may - - the guy who throws an his receiver should somehow be advised that we at home can read his - dick - corp. said it completed a restructuring agreement previously agreed to by the federal deposit insurance corp. creditor banks and subordinated debenture holders - the plan would permit the bank holding company to retire its bank and debenture obligations through exchanges of cash and equity - the fdic which in N provided $ N million in assistance to 's bank of oklahoma unit will continue to maintain $ N million in preferred stock in the bank unit - in exchange for the other $ N million the fdic will receive additional warrants it to buy N N of 's common stock outstanding up from the N N option the fdic received under terms of the N capital - in exchange for the $ N million they are owed creditor banks will receive N million shares of common stock and the proceeds from the future sales of four subsidiary banks to private buyers the bank holding company said - also under the agreement debenture holders will get one million shares of common stock in exchange for $ N million in debentures and holders of 's series a preferred stock will receive N shares of common stock for every share of preferred they own the company said - bear stearns 's chief economist lawrence in the sept. N issue of the firm 's global - were it true that a weak currency the way for trade surpluses then presumably argentina would be the center of today 's global economy - bsn corp. said it will begin an offer tomorrow to exchange up to one million of its common shares and all of its $ N million in N N N convertible debentures due N for a package of new debt and common stock warrants - under terms of the offer the sporting goods maker will swap $ N face amount of N N N subordinated notes due N and one warrant for each common share - each warrant allows the holder to buy one bsn share for $ N a share at any time over the next seven years - bsn currently has N million common shares outstanding - bsn also is offering $ N face amount of new notes and N warrants for each $ N face amount of its convertible debt outstanding - the company said it can redeem the warrants at its option for $ N each - the offer is n't contingent on a certain amount of debt or stock being exchanged - bsn said it is making the offer to shrink its capital and increase shareholder value - if all the bondholders and holders of one million common shares accept the offer bsn will increase its debt by $ N million but it also will recognize a $ N million gain from retiring the old debt said michael j. blumenfeld president - we have sufficient cash flow to handle that he said - the offers are scheduled to expire in to late november - merrill lynch & co. 's net income dropped N N while bear stearns cos. posted a N N gain in net and painewebber group inc. 's profit fell but would have risen without a special gain a year ago - at merrill lynch net was $ N million or N cents a share down from $ N million or N cents a share a year ago - total revenue reached $ N billion up N N from $ N billion - the firm 's drop in net reflected weaker revenue in transactions for its own account a decline of N N to $ N million on reduced revenue from trading fixed-income securities - investment banking revenue fell N N to $ N million on fewer equity and municipal - merrill lynch 's commission revenue grew N N however to $ N million on higher share prices and volume and on strong sales of mutual funds - revenue derived from interest and dividends jumped N N to $ N billion - fee revenue grew N N to $ N million - the brokerage also reported a loss of $ N million from the discontinued operations and disposal of its fine homes international limited partnership real-estate subsidiary - bear stearns said net in the first quarter ended sept. N reached $ N million or N cents a share from $ N million or N cents a share in the year-earlier quarter - gross revenue rose N N to $ N million from $ N million - profit from trading for its own account dropped the securities firm said - investment banking revenue climbed N N while commission revenue advanced N N on a stronger retail market - bear stearns is the holding company for bear stearns & co. the investment banking and brokerage firm - in new york stock exchange composite trading yesterday bear stearns shares closed at $ N down N cents - separately painewebber posted net income for the third quarter of $ N million or N cents a share reflecting a broad-based improvement in the company 's core businesses - retail profit surged but the company said it was only a modest to third-quarter results - a year ago net at the new york investment banking firm was $ N million or N cents a share including a special pretax gain of $ N million from the sale of the company 's interest in national car rental systems inc - revenue was $ N million including net interest down slightly from $ N million - in big board composite trading yesterday painewebber closed at $ N up N cents - corp. said it signed an agreement with martin to purchase its headquarters building the columbia center for $ N million - purchase of the structure is subject to execution of a definitive agreement approval by the boards of and its parent company bankamerica corp. and approval by regulators - the market upheaval apparently has n't triggered any cash crunch yet - individual investors investment firms and arbitragers who speculate in the stocks of takeover candidates can suffer liquidity and payment problems when stocks dive those investors often borrow heavily to buy their holdings and use the stocks as collateral for loans - but several large banks said yesterday they detected no signs of unusual demand for credit that would signal such difficulties - we 're seeing nothing out of the ordinary said one official at a top N bank - that 's good news because we all in this water - added another executive at a big bank we were all a little over the weekend trying to forecast what would happen monday but it 's been very quiet - now as for tomorrow hell who knows - what happened friday shows that financial markets are not yet sufficiently to handle another in prices - no with systems and procedures will ever prevent markets from suffering a panic wave of selling - but markets can operate with greater or lesser efficiency - after the N plunge markets agreed that it would be to halt trading whenever panic conditions arose - the new york stock exchange adopted two specific circuit breakers if the dow jones index falls N points in a day the exchange will halt trading for one hour if the decline hits N points the exchange will close for an additional two hours - the rationale is that an of trading will allow investors to reconsider their strategies calm sellers and lead buyers to enter the market at indicated new price levels - it is impossible to know whether that theory is realistic - a temporary of trading may indeed discourage a selling panic from feeding on itself - but there is also the possibility that down markets will intensify fears and cause an even more abrupt slide in prices - what happened friday was the worst of all - the futures exchanges followed their own circuit breakers and shut down at about N p.m. for N minutes after the standard & poor 's N stock index had fallen N points or about N points on the dow jones index - options markets stopped trading in many securities - the new york stock exchange under its own rules remained open - with nowhere else to go sellers and particularly program traders focused all their selling on the new york stock exchange - as liquidity on that market weakened prices fell sharply - had the futures and options markets been open additional liquidity would have been provided and the decline most probably would have been less intense - at N after intense telephone negotiations between the trading markets and washington the futures exchanges reopened - futures trading however was halted altogether at N after the futures markets had dropped an additional N points which is the daily limit for price declines - at this point the options markets also shut down and once more left all sales to be handled by the new york stock exchange - it is time to recognize that the new york stock exchange the futures markets and the options markets though separate have actually become so closely as to constitute one market effectively - traders can vary their strategies and execute their orders in any one of them - it therefore makes no sense for each market to adopt different circuit breakers - to achieve maximum liquidity and minimize price volatility either all markets should be open to trading or none - circuit breakers would not have halted the slide in prices on friday but they probably would have made for less volatile executions - it 's time for the exchanges and the securities and exchange commission to agree on joint conditions for trading or staying open - let 's not have one market shut down for N minutes when the dow declines N points and another shut down for an hour after a decline - the need for last-minute telephone negotiations among market officials will disappear once rules are in place that circuit breakers in all markets - the new circuit breakers if they are to be applied at all will require that futures and options trading continue as long as the new york stock exchange remains open - the rules should be established by agreement of the officials of all affected exchanges acting under the oversight and with the approval of the government regulatory agencies - should the sec and the commodities futures trading commission which with the sec the chicago stock-index markets be unable to agree the issue may have to be resolved by decision of the treasury secretary - in many ways our financial markets are better prepared today to handle a decline than they were two years ago - the new york stock exchange now has the capacity to handle a volume of nearly a billion shares a day - telephone service has been improved for customers trying to reach their brokers and specialists who i believe should stay despite the of some post-crash critics have larger capital positions - of course specialists ' actions alone can never prevent a major crack in stock prices - witness the fact that trading in some stocks closed early friday and opened late monday because of an excess of sell orders - but the task of improving market performance remains - mr. former chief economist of the new york stock exchange is a professor of economics at pace university 's business school in new york - a unified europe labor problems and prospects for u.s. firms - the social worker concerns of the european community 's plan to open its internal borders in N could set the effort off the if not done reasonably says general electric senior vice president frank doyle - u.s. companies wanting to expand in europe face tough pressure from unions in nations such as west germany which play a big consulting role in management decisions he says - corp. and international say unions also wo n't like plant and needed restructuring which means layoffs - many employers have already begun moving to southern countries such as spain and italy where wages are low and unions are weaker demand for trained labor and managers will rise there says - pfizer fluor and ge see big ec N a push for job training and ease in moving and finding workers - a fan was n't the baltimore ' fault - so said a federal judge in a case involving two players for the minor league va. a baltimore farm team - the players were by a during a july N N game with the - like its parent that year was not having a good year the judge said - after the game lost N three in the ninth he noted trouble began - more in the parking lot the players said led to a fight - the fan said he was and kicked by one player and that the other broke his with a baseball bat - the judge dismissed the fan 's suit against the team however ruling the innocent of hiring and not responsible for a fight that was outside the players ' employment - proposals arise for coping with the shortage of nurses - an association of academic health centers report urges nurses from duties that do n't require special skills - it also recommends better retirement and benefits and pay on education experience and nurses ' demanding work schedules - but it opposes an american medical association proposal for creating a registered care as potentially divisive it says the job would an unwanted new doctor 's extension - over a third of N hospitals surveyed by consultant associates use a clinical on performance and education - many also use recruiting bonuses tuition loan repayment or child-care help - some give incentives - systems signs up nurses for paid travel promising annual income up to $ N and free or subsidized housing - treating employees with respect is crucial for managers says consultant group after surveys of a million workers - it 's in their top five work values - fully N N of employees who say their bosses treat them with respect but only a third of those who do n't feel respected say they 're satisfied with where they work - up the digs about N employees of the maryland department of economic and employment development for four months painted walls and floors bought plants windows and and hung pictures at the agency 's baltimore office - the N hours of work will save the state $ N - curbing wage boosts will get high priority again in N collective bargaining a bureau of national affairs survey of N companies with next year indicates - despite warnings N N aim for wage increases of under N N and N N say they 'd try to replace workers if struck or would consider it - temporary workers have good the national association of temporary services says its survey of N such employees shows N N with more than a high-school education and N N with college degrees - about N N have retired from a full-time job while N N were asked to stay on full time - losses rise but they 're often covered by employers - but they search for ways to limit the damage - a third of N companies surveyed by the employee relocation council report a rise in N sales losses over N - about N N reimburse for all or some losses - since N more companies give aid as many real-estate values the council says - rjr nabisco pays up to $ N of losses including improvements - wo n't ensure loss coverage but will prevent a catastrophic loss it has given some employees the full purchase price when values fell from concern over dangers posed by a disposal site - federal express dow chemical ford and national city corp. will buy the home or let the worker sell to an outside firm but usually wo n't cover a loss - since N firms offering house to deter rose to N N of those the council polled from N N - the the national academy of engineering gives two of the semiconductor a $ N achievement award - now that 's letter carriers union president vincent philadelphia charles james of century management tactics - yesterday was in the words of new york stock exchange chairman john j. phelan jr. just your reasonably normal N up day - when it was all over and stocks had staged a huge recovery big board officials were about how well the day had gone - they said the exchange 's trading procedures personnel equipment and links with other exchanges could n't have performed better - we had no operating problems at all mr. phelan said after the market closed - all the things that we set up to slow down the process to let people know that the market was in an extreme position worked extremely well - prices for the N million shares that changed hands during the session were carried on the exchange 's trading tape with barely a delay officials said - while reaching blockbuster yesterday the volume was still well within the N capacity that the exchange has said it can handle daily since up its computers after the october N crash - the so-called circuit breakers devised by the big board and the chicago mercantile exchange to free falls in stock and futures prices were n't triggered yesterday because the markets were higher for most of the day - despite traders ' complaints mr. phelan said the links with the chicago futures market worked as planned in friday 's rout to provide a period - of greater help the big board chairman said was the natural circuit breaker of the weekend that provided a breathing period that brought back to the market - chicken chains by loss of customers - fast-food chicken chains faced with a worsening business slump are struggling to hatch some new marketing strategies - the crest report which tracks consumer purchases says customer traffic at chicken restaurants fell N N in the second quarter while the overall fast-food customer count was down N N - chicken business is off largely because of more competition from convenience food pizza and other fare says a spokesman for the report a publication of group a market research firm in port washington n.y - the loss of more customers is the latest in a string of problems - church 's fried chicken inc. and 's famous fried chicken inc. which have merged are still troubled by restaurant locations - chicken chains also are feeling more pressure from mcdonald 's corp. which introduced its this year and recently tested the sale of individual pieces of chicken - new management at kentucky fried chicken a unit of pepsico inc. has fought back with new medium and large chicken for the lunch crowd - and the chain is testing products that are n't fried such as chicken to try to win consumers - kentucky fried chicken also is testing of chicken which could be a hit with - but some fast-food industry analysts say problems with keeping chicken warm and fresh must be solved first - a kentucky fried chicken spokesman however disputed the notion that the delivery service experienced problems in some markets where testing has been discontinued - he says the test is continuing in chicago columbus ohio and a few other cities - the advertising industry is with rumors that kentucky fried chicken will drop young & rubicam and seek a new ad agency - but the company declines to comment - goldman a painewebber inc. analyst predicts kentucky fried chicken will post an N N drop in N net income - they 've been he says but they 'll have to become more aggressive - reluctant advertisers try spots - call it - pittsburgh consultant david bear is selling a soft approach to clients who want exposure yet ads - his radio spots that offer helpful hints - the only plug for the sponsor is a brief mention at the end of the spot - the messages resemble the business a daily of travel tips developed by mr. bear and sponsored by travel agencies in several major cities - new include burt hill associates a butler pa. architectural firm - its radio series features such spots as evening wear for urban structures and building a place to park - a harder sell says john the firm 's president would from the profession - hospitals have signed up to use the messages to promote and equitable gas co. is considering the format to offer energy tips to consumers - but such spots can be too soft - there 's always a risk of lost messages says john chairman of advertising usa which created similar radio spots for pittsburgh national bank - it 's a question of how much credibility you gain for the possible loss of recognition - retailer sees in environmental push - here 's a retailer that 's getting tough in the push for environmentally safe packaging and products - big bear supermarkets inc. a grocery chain based in san diego plans to display shelf cards and distribute recommending products deemed safe for the environment - the choices will be based on research by the san diego environmental health coalition and will include products like murphy 's oil soap and other - but the chain is quickly the of such - for example it recommends detergent and puts on its environmentally safe list - that does n't procter & gamble co. maker of cascade detergent - a company spokesman questioned the of the list noting that is present in all major - in fact bros. confirms that its brand does contain even though it is n't listed on the label for the version - thomas g. big bear 's executive vice president said the chain is still reviewing its product list to avoid such problems - our intent is to promote the best alternative he says - and it 's important that we be accurate - but in the end customers ' wishes are what will prevail - big bear does n't care for disposable which are n't - yet parents demand them - says mr. we 'll still be forced to sell items we might not agree with - odds and ends - does count at least in the grocery store - a study by 's marketing research shows soap sales climbed N N when bars were neatly on shelves instead of dumped in a wire basket - which celebrity are most - for the third year in a row consumers voted bill cosby first and james second in as spokesmen in tv commercials according to video tests new york - michael j. fox replaced bruce in third place placed fourth for the second time - health and human services secretary louis sullivan has chosen novello to be the next surgeon general bush administration officials said - if she is by president bush and confirmed by the senate dr. novello would succeed c. who rattled liberals and conservatives alike with his outspoken views on a range of health issues - dr. novello an expert on pediatric kidney diseases is deputy director of the national institute of child health and human development - she has also served on several task forces on acquired immune deficiency syndrome - dr. novello 's office said she would n't talk with reporters and it refused to release any information about her - the newsletter medicine & health which first disclosed her selection by dr. sullivan said she is N years old and she studied at the university of puerto rico school of medicine - the continuing series of hud scandals is a predictable result of pork-barrel politics - nevertheless such as the national association of home builders nahb continue to pressure capitol hill for more special-interest spending - kent nahb executive vice president argues that the u.s. faces a housing crisis reduced of homes for first-time buyers increased homelessness and lower apartment construction rates that will be very difficult to solve without expanded federal resources - there 's nothing unusual about business groups pushing for more government spending - but the nahb was created in N out of an organization that made its name fighting a administration proposal to take over all defense housing production - through the years the association has been an active member of the taxpayer 's coalition pushing for such initiatives as the amendment - yet on matters close to home - the hud budget has dropped by more than N N since N argues mr. - we 've taken more than our fair share - i would n't have a problem if other programs had taken a similar hit - but nahb support for subsidies is not related to the current housing crunch over the years the nahb has backed a host of public programs - it once pushed for a national housing production goal set by the federal government and has regularly advanced housing measures - moreover explains one hud official the nahb remains susceptible to internal pressure from members that specialize in subsidized production - the association is pushing an extensive and expensive which would substantially boost spending above the current level of more than $ N billion annually - direct federal subsidies for housing construction have proved expensive in the past and inevitably are to the benefit of developers and lobbyists as demonstrated by the ongoing hud scandal or congressmen - indirect subsidies through the fha for instance are little better - though mr. says expanding fha lending would result in no cost to the government the mere diversion of funds from other parts of the economy and from other forms of housing such as low-income to the single-family home market would result in a major expense - more important housing programs run by hud the va and are in red ink - the fha alone lost $ N billion in fiscal N the government 's equity in the agency essentially its reserve fund fell to minus $ N billion - the federal government has had to pump in $ N billion into the va housing program since N to keep the fund afloat and the va requested an additional $ N million for the fiscal year just ended - all told the federal government already guarantees more than $ N billion of mortgages - in its produced publication where will our children live the nahb does acknowledge that of course the full measure of housing can not be provided by the federal government - it points to the impact of local government regulation particularly and building fees which the price of housing out of the reach of and people - but while the nahb has suggested actions that states and should take to reduce regulatory barriers the association has proposed no activist legislative program comparable to say its detailed request for more federal subsidies to eliminate controls - the association a majority of whose N members build fewer than N units a year is like many other business - explains macdonald of the national taxpayers union it in two - the builders like the subsidies but at the same time they tend to be fiscal conservatives in terms of major issues such as the amendment - unfortunately the organization 's desire for pork tends to override its commitment to overall fiscal responsibility - two years ago when the nahb lobbied for the $ N billion omnibus housing bill the organization basically dropped out of the taxpayers ' coalition says ms. macdonald - as mr. of the nahb acknowledges government is not going to solve the problem - the real key is to have the economy working and interest rates down - more money for hud will increase the deficit and the economy more money to municipalities that are their local housing markets will further them from the effects of their policies - is this what the home builders want - mr. is a institute fellow - see related story and bills to make wishes come true wsj oct. N N - in an attempt to give new momentum to european community plans for a single currency ec government leaders are likely to agree to set a date for starting formal talks on the ec 's founding treaty of rome - according to diplomatic sources in brussels most ec leaders agree that talks should begin in the second half of N and will make a declaration on that during a summit meeting in france on dec. N and N - the only strong opposition to changing the ec treaty comes from british prime minister margaret thatcher who is opposed to creating a single ec currency - but the process of the conference does n't require - setting a date to start treaty negotiations has no legal significance in itself but could be viewed as an important psychological push - french president mitterrand fought to set a date for the conference during the ec summit in madrid last june but the move was because of opposition by mrs. thatcher and west german chancellor helmut kohl - diplomatic sources said mr. kohl may now agree to set a date for the conference to make it clear that west germany is still committed to ec unity - the latest in the equities markets me of the joke t. boone pickens tells about the guy who was run over by the parade - when asked what went wrong the unfortunate victim replied it was a combination of things - and so it was on gray friday - the grand of this parade would appear to have been excess leverage - even if that is so however it 's probably the case that no barriers should have been to stop the before the end of the rout e - the began friday afternoon when word spread that the ual buy-out was - although the expects to patch together a substitute offer consisting of less cash the failure to get cash from japanese and american banks confirmed a growing fear among arbitragers that the of takeover deals is ending - lots of other made up the parade of course notably a surprisingly large increase in producer prices federal reserve and the bush administration 's temporary defeat in trying to lower the capital-gains tax - as usual few favorable reviews were heard for that band of program traders although most serious studies suggest they only play the music that others write - what really spooked the along wall street however was the sudden concern that whatever the reason the pool of debt capital is up - gray friday reflects a panic mainly by the takeover arbitragers rather than the small investor as their highly investments in the deal stocks are by the unexpected up of the for deal financing - deal stocks led the market down as they absorbed the heaviest losses - ual which triggered the slide opened monday at $ N down about N N from thursday 's close - amr opened monday at $ N down nearly N N from thursday 's close - both took further hits yesterday - hilton lost N N on friday paramount lost almost N N - a careful look reveals that where deal financing has been secured the target 's stock price was not affected on friday - the multibillion-dollar prospects where the bidder must line up a consortium of banks issue billions in high-yield debt were where the damage was concentrated - the market for so-called junk bonds has been setting the stage for friday 's dramatic march for several weeks - the growing financial difficulties of recent restructurings or takeovers such as resorts international integrated resources and campeau 's retailing empire have cast a pall over the entire market for high-yield securities - investors have reacted by ignoring recent efforts to float junk bonds by ohio and by forcing ramada to postpone indefinitely its planned junk-bond sale and restructuring - as a result high-yield mutual funds have declined across the board and the many firms planning to sell $ N billion in junk bonds before year-end are experiencing anxious times - these are all market excesses putting aside the boosts that the tax code gives to debt over equity and what we 've seen is the market them in - of course washington had n't been silent in the days leading up to the debacle and its tendency to in the leverage equation remains a troublesome prospect but those preliminary steps should n't us from the basic market that was at work on friday - if it is correct to find that concerns over corporate debt and lbos caused gray friday what are the implications for policy makers - after all the stock market 's response to the collapse of the ual deal might be taken to confirm the direction of regulators - is this a case where private markets are of washington 's of wall street - absolutely not - to the extent that friday 's sell-off reflected a sudden of the excesses of leverage the message is that wall street and the private markets are fully capable of imposing the appropriate incentives and sanctions on corporate behavior - the national economic interests are much better served allowing the private interests of bankers and investors be the ultimate judges of the investment quality of various lbo deals and leveraged restructurings - the recent difficulties in the junk-bond markets and the of bank capital for recent deals the wisdom of letting the free markets operate - if takeover premiums become excessive if lbo become too aggressive then the private market will recognize these problems more quickly and accurately than will policy makers and the markets will move with speed to impose appropriate sanctions - yes the broader exchanges got caught up in the but they rode the tiger up all year - not surprisingly he sometimes - the arbitragers and takeover got killed on gray friday while the besieged managers of prospective targets cheered - if you identify with the besieged managers you must concede that and effective relief from the excesses of the takeover market is more likely to come from the marketplace than from washington - if you side with the arbitragers and raiders you clearly have more to fear from private investors than from regulators although the delaware courts should never be underestimated - the truth is washington understands politics better than economics - although the average citizen is probably not too much from washington 's war against wall street regarding excessive financial actual legislation would probably impose considerable harm - any such attempt to good debt from bad debt or to draw the line at a particular industry such as the airlines is likely to blunt the spur that the proper amount of leverage provides both to equity markets and economic efficiency in general - far better for policy makers to concentrate on the war against drugs panama and the deficit all of them that seem never to end - mr. former top economist at the securities and exchange commission teaches at the university of rochester 's simon business school - tokyo share prices rebounded tuesday morning with the nikkei index of N selected stocks rising N points to close the morning session at N - the index slid N points or N N on monday - in the first N minutes of tuesday 's trading the nikkei index soared N points to N - by N a.m. tokyo time the index was up N points to N as investors hailed new york 's overnight rally - monday 's slide came in a relatively calm session that did n't provide much direction for other markets - shares also closed sharply lower across europe particularly in frankfurt although london and a few other markets recovered some ground after stocks began to rebound in new york - other asian and pacific markets had sharper losses than tokyo but the selling wave stopped short of another market crash - all eyes were on tokyo at the opening because it was the first major market to trade since friday 's 190.58-point plunge on wall street - but rather than set the tone for other markets japan 's major institutional investors chose to remain on the sidelines - still despite the sudden of stock-market turbulence managers of japanese investment funds said they were n't planning to unload u.s. or european equities - we did n't trade much today as our policy now is to wait and see said a fund manager at life insurance co - we would like to wait and see until trading goes around through europe and new york - the institutions appeared confident that japanese regulators would step in to ensure orderly trading if necessary and there was considerable speculation during the day that the finance ministry was working behind the scenes to do just that - but in the absence of trading its presence was never felt - at the close the nikkei average of N stocks stood at N down N points or N N - the broader tokyo stock price index sank N or N N to N - the day 's decline was generally in line with analysts ' weekend predictions - declining issues advancers N - but volume was thin at N million shares compared with N million friday - the market opened sharply lower with the nikkei average down nearly N after N minutes - a midmorning rebound brought it back to show a gain of about N at the end of the morning session but the rally failed in the afternoon and the market closed near the day 's low - the smaller stocks in the tokyo market 's second section also posted their biggest decline of the year - the tokyo stock exchange index for the second section fell N or N N to N - many investors trying to outperform the market 's major indexes have to these small issues in recent weeks - japanese investors and traders expressed relief that the tokyo market did n't fall more sharply - but its performance did bear some to events of two years ago during the october N global stock market crash - on oct. N N the friday before the black monday crash the new york market dropped N N and tokyo followed on monday with a N N drop - this time wall street 's plunge of N N friday was followed by yesterday 's N N loss in tokyo - two years ago tokyo 's biggest fall came the day after new york 's N N black monday plunge when the nikkei average fell N N - thus market participants yesterday were looking ahead nervously to wall street 's opening - but in new york yesterday the dow jones industrial average surged N to close at N on heavy volume of N shares although declining issues still outnumbered advancing ones on the broad market - a director at yamaichi investment trust & management co. called yesterday 's session a good scenario for japan - now we are looking for the time to place buy orders he said - for us institutional investors the chance for buying has come - general manager of the investment research department at trust & banking co. also was optimistic - he described friday 's plunge in the u.s. as a fleeting event resulting in part from excessive merger and acquisition activity - unless there is a panic this is the best time to buy as was the case two years ago he said - those shares which had posted gains on speculation were dashed with cold water but as far as major stocks are concerned there is n't much impact - other fund managers were similarly - we have no plans to adjust our asset allocation in foreign equities said chief portfolio manager in the pension fund management department at trust & banking co - he said friday 's wall street decline was well within the range of volatility that trust plans for when it charts its overseas investment strategy - among other asian and pacific markets malaysia and singapore had the biggest losses with the kuala lumpur composite index in malaysia falling N N and singapore 's times industrial index down N N - major indexes declined more than N N in australia and new zealand and N N in hong kong - manila seoul taipei and escaped with slightly smaller losses - brokers and fund managers said the region 's markets were reacting to friday 's wall street plunge even though that decline was due to local factors such as failed corporate buy-outs and a deteriorating junk-bond market - it 's pure psychology said william an account executive for drexel burnham lambert ltd. in hong kong - markets in this region are n't so geared to leveraged buy-outs and their economies generally are in good shape but there 's no doubt that asia is still following america 's lead - several analysts said malaysia and singapore had the biggest losses because they are relatively open to rapid cash flows - hong kong is the region 's next most open market but many foreign investors have been staying away from it since it plunged in june amid political turmoil in china - singapore took the hit because when people want to get out they tend to go where the liquidity is said elizabeth hambrecht a regional analyst with baring securities hong kong ltd - she pointed out that even after monday 's N N decline the times index is up N N this year so investors who out generally did so profitably - similarly kuala lumpur 's composite index yesterday ended N N above its N close - in hong kong the hang seng index fell N to finish at N - trading was heavy at about one billion shares compared with N million friday - but the session was orderly in contrast to the market 's four-day after the N crash - richard a director at hong baring international fund managers ltd. said the market probably has n't hit bottom yet but is close - if new york does n't collapse i see maybe another N N on the downside not counting the risk of bad news out of china he said - in australia sydney 's all index closed at N down N N its biggest drop since october N - but volume rose only to N million shares from N million friday - an analyst at brokerage firm & young ltd. described the market 's performance as as investors fled to australian stocks and entrepreneurial companies they perceived as having any takeover premium built into the price - london 's financial times-stock exchange 100-share index the most closely watched market barometer ended at its intraday high of N down N or N N - at its low shortly before wall street opened it was off more than N points - the financial times 30-share index closed N points lower at N - volume more than doubled to N million shares from N million friday - prices on the frankfurt stock exchange tumbled in heavy trading - the decline in the german stock index of N points or N N to N was the frankfurt market 's fall ever - retail investors dumped holdings on a massive scale pushing some blue-chip shares down as much as N N - analysts cited memories of two years ago when many small investors held on to their shares after the october crash but the west german market continued to decline for the next three months - here are price trends on the world 's major stock markets as calculated by morgan stanley capital international perspective geneva - to make them directly comparable each index is based on the close of N equaling N - the percentage change is since year-end - frank lloyd wright is reported to have said once that if you the world on its side everything loose would end up in california - we 've always thought that mr. wright underestimated california 's but maybe the state 's are starting to the forces that made it such a significant place - what else is one to make of the initiative just proposed by several major environmental groups and organized by the state 's attorney general - if passed by the voters the recently announced initiative would phase out major pesticides reduce carbon dioxide emissions by N N ban new offshore drilling ban chemicals thought to the ozone layer and create a new state environmental officer armed with a $ N million budget to sue any firm or agency he thinks is being too - the initiative is based largely on the of the green lobby the sierra club the league of conservation voters the natural resources defense council the national campaign and the citizens for a better environment - the environmental defense fund is having nothing to do with this one - not only californians but all americans would pay if this thing passed - the initiative bars the sale of any crops in california that do n't meet the initiative 's standards - kansas wheat farmers and florida fruit growers would have to adjust or give up the california market - in other words california is to take control of the nation 's farm policy - as usual the green lobby 's proposal is from scientific reality - consider the provision - the proposed initiative would mandate a reduction of carbon dioxide of N N - even if one buys into the whole greenhouse theory it is that reductions in a single state could have any impact on what is billed as a global problem - but if rational science and economics have nothing to do with the new environment initiative what is going on - the first place to look under these circumstances is at the ways in which the sponsors themselves will benefit - the key here is the of state attorney general john van de - he 's running for governor - mr. van de is the one who collected the plans from the various radical environmental groups and them into a single initiative to be placed on the ballot for election on nov. N N - that 's also the day of the gubernatorial election - the initiative seems to have been to include all the hot issues that set off the wealthy hollywood who money - and it allows mr. van de to get around campaign spending limits - he can spend the legal maximum for his campaign all the spending for the van de initiative on which there are no limits is - this initiative is being labeled the big green but maybe it should be called the big - the republican candidate sen. pete wilson is playing the initiative game too his own crime initiative - while it is possible that the big green initiative will be ruled unconstitutional it is of course that in modern california it could slide through - this is the state that recently passed the N initiative - if this new proposal ever does become law the green lobby will benefit directly - the initiative creates a free floating state environmental officer to sue companies or government agencies that do things he does n't like - that means the and such groups no longer would have to spend as much money on litigation taxpayers would bear the cost - mr. van de and his allies may be hoping that the environment is such a mom and issue among certain segments of california 's population now that almost any collection of nonsense can pass under its - of course the state 's liberals are not yet a nation themselves - george bush for example may decide that he does n't want to be the president who lost control of interstate commerce to an attorney general from california - and some other segments of california 's political and media culture may yet start to point out that the initiative would impose significant costs on the state 's less affluent citizens in the form of higher food prices and lost jobs - this initiative will help california define itself for the future either as a state still to economic and scientific reality or as one being led to wherever its activists want to take it - first there was a death watch - then - spurred by waves of large-scale buying in blue-chip stocks the dow jones industrial average rallied yesterday and erased about a half of friday 's 190.58-point plunge gaining N to N - it was the advance for the average of N blue chips on new york stock exchange volume of N shares the highest since the days after the N crash - while the advance cheered investors who feared a crash would occur yesterday it was strictly a rally fed by huge buying by bargain-hunting institutions and program traders - a troubling sign declining stocks on the big board outnumbered advancers N to N and the over-the-counter market that includes many smaller stocks suffered aftershocks of friday 's late big board plunge - the nasdaq otc index closed down N to N - meanwhile in a divergence in two of the market 's most important indicators the dow industrials ' sister average the dow jones transportation average tumbled N to N its decline next to the fall during the N crash - plunged on takeover disappointments in two airline stocks ual and amr which each fell more than N N when they reopened for trading yesterday after being suspended friday afternoon - ual the takeover stock at the center of friday 's 190.58-point market plunge fell N N to N N on nearly N million shares - overall this is a rally but it 's very selective said arthur jr. a veteran painewebber inc. trader at the big board - everyone was a little concerned about the general of the rally and failure of the otc market to get into plus territory - it 's just a strange feeling - i do n't think anyone left the place - the rally gave at least for now to the declaration of big board chairman john j. phelan jr. that friday 's market debacle was an condition and not a disaster - but to traders it looked like disaster on the N a.m. opening bell - the dow jones industrial average opened down N shortly after N - but most of the N blue-chip stocks in the average including eastman kodak and general motors could n't trade because of the heavy backlog of sell orders left over from friday 's rout - at N procter & gamble one of the most important dow of late opened down N N to N - the dow dropped to a quick loss and to many traders it looked as if stocks were headed for yet another big tumble - more stocks opened over the half hour as the N big board specialist firms in charge of keeping the market orderly to find buy orders from major brokerage firms to match the selling flood - then to make matters worse computerized sell programs kicked in stocks into losses - there was heavy stock-index arbitrage as traders sold big baskets of stock and bought stock-index futures to profit from the price discrepancies between the two markets - this was a from friday when standard & poor 's 500-stock index futures had closed at a sharp discount to stocks - the of the program selling dashed any hopes that some of the big program trading firms would hold off until the market stabilized - they did n't - the dow accelerated its slide losing N in the first N minutes of trading - with program traders seemingly in charge buyers backed away from the market and watched stocks fall - then at N the dow suddenly started to rebound and when it shot upward it did so even faster than the fall - and this time it was n't just the program traders who were responsible - all the selling had pushed stocks to such cheap values that big investment banks and major money management firms started buying stocks heavily - the program traders were in there too of course - but according to one trader the programmers did n't look as dominant on the upside as on the downside because there was also a lot of bargain-hunting by institutions - m. director of the new jersey division of investment which oversees $ N billion in investments said the first thing we did was to double our orders yesterday morning - with the market down like this we 'll probably take another $ N million and put it in the market - trading in walt disney co. particularly caught traders ' eyes - according to big board officials disney had one of the biggest imbalances on friday it was one of the seven stocks that could n't finish trading that day - the stock opened late at N N down N N - but then it shot upward N N as goldman sachs & co. stepped in and bought traders said - however disney specialist robert said i would be surprised if goldman represented N N of the opening volume - around wall street trading desks were relieved that they could at least play the market yesterday in contrast to friday 's gridlock - at donaldson lufkin & jenrette inc. head equity trader said i think the opening was - it was orderly - we put some orders together - there was n't a lot of panic selling either domestically or internationally - not like friday where they just took the market apart - still the market had n't yet crossed into positive territory and traders were - but in another dramatic burst the dow tacked on N points in five minutes and at N the index showed a gain of N - on the big board floor and on trading desks traders their approval - peck a trader in shearson lehman hutton inc. 's otc department i tell you this market acts healthy - around him scores of traders seemed to get a burst of energy their boss broke out bottles of water to cool them off - among big board specialists the cry was pull your offers meaning that specialists soon expected to get higher prices for their shares - it was on the upside said one big board specialist - but not everybody was making money - the on the chicago board options exchange the nation 's major options market was heavy after the trading in s&p N stock-index options was halted friday - many market makers in the s&p N index options contract had bullish positions friday and when the shutdown came they were frozen with huge losses - over the weekend clearing firms told the chicago market makers to get out of their positions at any cost monday morning - they were absolutely killed said one chicago-based options trader - some traders said that the closely watched major market index whose N stocks mimic the dow industrials did n't lead yesterday 's big rally - james a partner at specialist & said the difference between today and two years ago terrible tuesday oct. N N is that then we needed a to go into the major market index spend $ N million and get the program rally started - this time institutions saw the programs coming and backed away and backed away - then when the market was at a technical level to buy they came in with a - however according to one analyst the timing of major market index futures buying just before the turnaround was similar to that of terrible tuesday - futures were pulling the stock market higher said donald head of stock-index futures research at prudential-bache securities inc - although the big board 's specialist firms struggled through another highly volatile trading session their performance yesterday was better than during friday 's chaos according to traders and brokers who work with them - specialists were criticized for their inability to maintain orderly markets during the friday plunge - but yesterday even with halts in such major blue-chip stocks as merck we expected the halts and it was n't too bad said donaldson 's mr. who had been critical of the specialists ' performance on friday - according to a big board official while many stocks opened late there were subsequent trading halts in only three issues amr merck and energy - merck is one of the most important stocks in the major market index - no sector of the market has been during the past two days ' gyrations - yet from the dow industrials ' high on oct. N through friday 's plunge relatively good performances have been turned in by real-estate utilities precious metals and life insurance stocks - and yesterday the top performing industry group was oil field equipment issues - for example jumped N N to N rose N N to N N and baker hughes rose N N to N - because of the ual and amr airlines were the weakest sector of the market yesterday - philip morris was the big board 's most active issue rising N N to N N on nearly eight million shares - among other major issues coca-cola co. closed up N at N N on N million shares and american telephone & telegraph rose N N to N on nearly N million shares - shares of international business machines which reported earnings yesterday finished at N up N after slipping below N during friday 's session for the first time in five years - shares of three brokerage firms rose after they reported earnings - merrill lynch added N N to N painewebber rose N to N N and bear stearns rose N to N N - federal national mortgage association a recently hot stock climbed N to N on nearly N million shares - at a news conference after the close of trading yesterday the big board 's mr. phelan and other exchange officials praised the performance of their computers and personnel - mr. phelan said that program trading strategies were n't responsible for triggering friday 's decline despite a jump in the use of the computer-driven strategies in recent months - some N million of the more than N million shares traded in the final N minutes of friday 's session when the plunge in stock prices was concentrated were he said - program trades make up N N of the exchange 's volume on an average day but despite the increase friday it was certainly not something you would say the market decline mr. phelan said - mr. phelan expressed relief that the market rebounded yesterday - obviously every time we get this kind of reaction it 's going to make everybody nervous including me he said - he said that exchange officials had conversations with wall street firms throughout the weekend and that all the participants behaved very very today - meanwhile peter dapuzzo shearson 's head of retail equity trading praised institutional investors in the otc market who were heavy buyers of the nasdaq 's biggest technology issues yesterday amid a flood of selling by other investors - the institutions ca n't be criticized for their behavior mr. dapuzzo said in an interview - it was the opposite of what happened on oct. N - they used their judgment - they did n't panic during the first round of selling this morning - instead they bought on weakness and sold into the strength which kept the market orderly - maybe they learned from experience - mr. phelan said the performance of specialists during friday 's plunge was because out of N big board common stocks traded during the day only seven were closed and were n't reopened before the close - they did an excellent job mr. phelan said of the specialists - wall street traders on friday had complained about the trading - james a. white and contributed to this article - west germany 's green party joined its ideological and the institute in the legal battle to ground the atlantis shuttle and its galileo probe to jupiter - the greens wanted a washington federal appeals court to block today 's scheduled long enough for them to ask the world court to order a permanent cancellation of the $ N billion flight - a appeals panel yesterday refused to comply though liberal judge pat went out of her way to deny that this was a case - of course it was - nasa should now sue for fines against all three foreign and domestic for bringing this case - a house-senate conference approved a permanent smoking ban on all domestic airline routes within the continental u.s. and on all flights of six hours or less to alaska and hawaii - the restrictions would cover all but a small percentage of domestic air traffic and represent a major expansion of the current smoking ban on flights of two hours or less - the exemption allowed on longer flights to alaska and hawaii appears to be largely a for the traditionally powerful tobacco industry which has found itself increasingly isolated in the face of public pressure in recent years - by a N margin house negotiators initially rejected last night a senate provision covering all domestic flights - but the compromise was soon agreed to in subsequent discussions - as a practical matter flights from the west coast to hawaii would be covered as they are under the time limit but the language would exempt longer routes beginning for example in chicago or on the east coast - within the senate the ban has had aggressive support from sen. frank d. n.j. who has used his position as a senate appropriations subcommittee chairman to votes for the initiative - the measure is attached to the more than $ N billion fiscal N transportation bill within mr. 's jurisdiction and the final compromise is with more than $ N million in road projects earmarked by members as well as funds sought by major airports including denver - from the outset the tobacco industry has been uncertain as to what strategy to follow - but the industry retains support in the house leadership through the influence of grower states such as north carolina - majority whip william gray owes a political debt to southern agriculture lawmakers for his rise in the house and the philadelphia democrat used his position in the conference to salvage the exemption from a total ban - although the smoking provision has attracted the most public interest the underlying bill was the subject of lobbying because of its impact on air transportation and the more mundane but politically important projects of members - in a stark lesson in the power of the appropriations committees the house deliberately killed a handful of projects backed by lawmakers in florida illinois and pennsylvania who had voted against the panel leadership on the house floor - anybody can vote as they want said rep. william lehman d. fla. head of the house conferees - but if you make a request you should support the committee - within the federal aviation administration the final bill promises to increase spending for facilities and equipment by more than N N from last year and total operations would rise to $ N billion a N N boost - the facilities account includes $ N million for denver 's ambitious new airport and the competition for these funds created shifting alliances between urban lawmakers representing established airports in philadelphia and michigan and the major carriers to denver united and continental - leery of the costs and critics say competition the airlines have sought to gain leverage over the city of denver - texas air corp. which owns continental and the air transport association were prominent in the lobbying - the industry sought to impose conditions that would have delayed funds for the project until denver and the airlines had agreed to leases for N N of the gates - but this was rejected in favor of much language the transportation department to review the costs of the first phase expected to cost about $ N billion - though smaller in total dollars the conference agreed to preserve an estimated $ N million in controversial subsidies to carriers serving rural or isolated airports - the sum is more than double what the house had approved for the program but the list of qualified airports would be cut by N under new distance requirements and limits on the level of subsidy - congress previously cut six airports this year - the impact of the changes is to eliminate many of the most excessive cases where the government has been paying more than $ N for each passenger in subsidies - among rail and highway accounts the agreement provides $ N million for including $ N million for capital improvements - and grants for mass transit would be effectively frozen at $ N billion or $ N million more than last fiscal year - enjoying several blockbuster movie hits including batman los angeles-based guber-peters entertainment co. reported earnings for the first quarter ended aug. N of $ N million or N cents a share compared with a year-earlier loss - sony corp. which has offered to acquire the company is seeking to free its top executives peter guber and jon peters from an exclusive agreement with time warner inc. 's warner communications inc. so they can run columbia pictures entertainment inc - sony two weeks ago agreed to acquire columbia for $ N billion or $ N a share - warner sued sony and guber-peters late last week sony and guber-peters have charging warner with attempting to interfere in sony 's acquisition of the two companies - guber-peters 's net income in the latest quarter compared with a net loss of $ N million or N cents a share in the year-earlier period - the company said revenue rose N N to $ N million from $ N million reflecting the success of its movies in the and as well as the batman - a group including jon m. of salt lake city said it boosted its stake in chemical corp. to N N of the the common shares outstanding - as previously reported holdings corp. owned by jon m. and other members of his family proposed that corp. an affiliate of holdings acquire in a friendly transaction for $ in cash or $ N million - in a filing with the securities and exchange commission the group said it controls N common shares including N shares bought from aug. N to oct. N for $ N to $ N per share - officials at based in pittsburgh declined comment - congress has been critical of the bush administration for not sending enough aid to poland so it is getting ready to send its own version of a care package - last month the senate voted to send a delegation of congressional staffers to poland to assist its legislature the in democratic procedures - senator pete calls this effort the first gift of democracy - the poles might do better to view it as a horse - it is the vast shadow government of N congressional staffers that helps create such legislative as the N page reconciliation bill that claimed to be the budget of the united states - maybe after the staffers explain their work to the poles they 'd be willing to come back and do the same for the american people - plc a financially troubled irish maker of fine crystal and china reported that its pretax loss for the first six months widened to N million irish punts $ N million from N million irish punts a year earlier - the results for the half were worse than market expectations which suggested an interim loss of around N million irish punts - in a sharply weaker london market yesterday shares were down N pence at N pence N cents - the company reported a loss after taxation and minority interests of N million irish punts compared with a loss of N million irish punts for the year-earlier period - there were n't any extraordinary items - sales for the total group rose N N to N million irish punts compared with N million irish punts a year ago - has decided against paying an interim dividend - said the appointment of a new management team and the signing of a comprehensive labor agreement are expected to enhance the company 's long-term prospects - the sudden flight to quality that triggered friday 's explosive rally was reversed yesterday in a flight from quality rout - the setback in which treasury bond prices plummeted reflected a rebound in the stock market and profit-taking - it was a pretty wild day - our markets were closely tied to the stock market said joel manager of trading at smith barney harris upham & co - friday 's flight to quality was no longer needed once the stock market found its he said - some fixed-income investors had expected a further drop in stock prices after the nearly drop in the dow jones industrial average on friday - that caused investors to stocks and buy high-quality treasury bonds which are safer than other types of securities - but when stocks began to climb instead prices of treasury bonds declined - contributing to the selling pressure were by several investment firms advising clients to boost their stock holdings and reduce the size of their cash or bond portfolios - among the firms were merrill lynch & co. and dean witter reynolds inc - the bond market seemed to ignore evidence that the federal reserve eased credit conditions slightly by allowing the federal funds rate to as low as N N N - the closely watched rate on federal funds or overnight loans between banks slid to about N N N last week down from its perceived target level of about N N - the rate is considered an early signal of changes in fed policy - traders said yesterday 's modest easing did n't stir much enthusiasm because it had been widely expected - in fact some economists contend that the latest easing started last week - others note that some investors were disappointed because they had expected a more aggressive easing - the treasury 's benchmark 30-year bond ended about N N points lower or down about $ N for each $ N face amount - the reversal was even more evident among treasury securities - after treasury bill rates plummeted as much as N percentage point on friday they gave back of that amount yesterday - the bond-equivalent yield on three-month treasury bills for example was quoted late yesterday at N N compared with N N friday - investment-grade corporate bonds mortgage-backed securities and municipal bonds also fell - but prices of junk bonds which were battered friday in near standstill trading rebounded to post small gains after a volatile trading session - junk bonds opened as much as four points lower but staged a modest comeback as stock prices firmed - some traders said the high-yield market was helped by active institutional buying - in particular they said firms such as first boston corp. and drexel burnham lambert inc. began making a market in junk issues early in the session when prices hit severely depressed levels - i think the willingness of securities companies to make markets for high-yield issues improved the sentiment for junk bonds said john an economist at moody 's investors service inc - u.s. treasury bonds were higher in overnight trading in japan which opened at about N p.m. edt - the benchmark 30-year bond for example rose one point in early japanese trading in reaction to a quick drop in the tokyo stock market - but as japanese stocks rebounded treasurys retreated and ended just modestly higher - many u.s. trading operations wanting to keep a eye on japanese trading as an indication of where u.s. trading would begin were fully during the tokyo trading session - most of the action was during the night session said michael moore trading manager at continental bank - jay who often trades overnight for capital insight inc. beverly hills calif. said trading in tokyo was very active but highly volatile - we went down N point in N minutes right before lunch then after lunch we went up N point in N minutes he said - in tokyo trading is halted during - tokyo 's market turned out to be a bad bellwether for u.s. trading - when the market opened here bonds prices fell as the stock market regained strength - the bond market 's focus on stock activity was so strong yesterday that it today 's slate of economic data which includes the government 's report on august u.s. merchandise trade and september industrial production - industrial production is expected to have declined N N according to a consensus of economists surveyed by dow jones capital markets report - the august trade deficit is expected to have widened to $ N billion from $ N billion in july - a widening of that magnitude said one new york trader is not a favorable number - it could do damage to us - meanwhile agency supply is expected to weigh heavily on the market today when the federal home loan bank prices a $ N billion offering of one-year three-year five-year and 10-year maturities - tomorrow the resolution funding corp. will provide details of its first bond issue which is expected to total between $ N billion and $ N billion and carry a maturity greater than N years - resolution funding is a division of resolution trust corp. the new federal agency created to bail out the nation 's troubled thrifts - and this week the tennessee valley authority plans to price a $ N billion offering its first public debt borrowing in N years - there 's lots of supply the new york trader said - we have a couple or three tough weeks coming - treasury securities - prices of treasury bonds tumbled in moderate to active trading - the benchmark 30-year treasury bond was quoted late at a price of N N compared with a closing price of N N friday - the yield on the benchmark issue rose to N N from N N - the latest 10-year notes were quoted late at N N for a yield of N N compared with N N to yield N N - short-term interest rates fell yesterday at the government 's weekly treasury bill auction - the average discount rate on new three-month treasury bills was N N the lowest since the average of N N at the auction on oct. N N - the average discount rate was N N on new six-month bills the lowest since the average of N N at the auction on july N N - here are auction details - rates are determined by the difference between the purchase price and face value - thus higher bidding narrows the investor 's return while lower bidding widens it - the percentage rates are calculated on a year while the yield is based on a year - both issues are dated oct. N - the 13-week bills mature jan. N N and the 26-week bills mature april N N - corporate issues - investment-grade corporate bonds ended one to N N point lower - there were no new issues - foreign bonds - foreign bonds surged as the dollar weakened against most major currencies - among benchmark issues japan 's no. N N N bond due N ended on brokers screens at N up N point - the yield was N N - west germany 's N N N issue due june N ended at N up N point to yield N N - britain 's N N N bond due N ended N N higher at N N to yield N N while the N N N notes due N rose N to N N to yield N N - mortgage-backed securities - mortgage securities gave up most of friday 's gains as active issues ended N to N point lower - dealers said morning activity was hectic as prices dropped in response to gains in the stock market and losses in treasury securities but trading slowed to moderate levels in the afternoon - government national mortgage association N N securities for november delivery were quoted late yesterday at N N down N from friday N N N securities were down N at N N and N N securities were at N N off N - federal home loan mortgage corp. N N securities were at N N down N - on friday mortgage issues gained as much as N N - late yesterday ginnie mae N N securities were yielding N N to a 12-year average life assumption as the spread above the treasury 10-year note narrowed N percentage point to N - traders said there were some busy dealings in freddie mac and federal national mortgage association securities because underwriters from last week 's heavy slate of real estate mortgage investment issues moved to gather collateral for new deals - offsetting the purchases were continued heavy sales by mortgage which are producing increased amounts of fixed-rate mortgage-backed issues with lower rates - there was no new-issue activity in the derivative market - municipals - rebounding stocks and weaker treasury prices drove municipal bonds N to N point lower in late dealings - the session losses left municipal dollar bonds close to where they were before the 190.58-point drop in the dow jones industrial average friday prompted a capital markets rally - trading was hectic during the morning with players trying to gauge whether equities would continue friday 's free fall or stabilize after a brief spot of weakness - started the session flat to a touch higher on anticipation of further stock market erosion but bond prices rapidly turned south as it became more clear that a repeat of the october N crash was n't at hand - professionals dominated municipal trading throughout the session - traders said retail investors seemed to be the sidelines until a measure of volatility is out of the market - new jersey turnpike authority 's N N issue of N was off N at N N bid yielding N N up N percentage point from late friday - florida board of education 's N N N issue of N was N point weaker at N N bid - the N N N issue of bridge and tunnel authority of new york due N was off N at N N bid - and county va. water authority 's N N N issue of N was down N at N N bid - serial bond yields were up about N percentage point - corp. kansas city mo. said it 's weighing strategic alternatives for its business men 's assurance co. unit and is possible buyers of the life and health insurance operation - a spokesman said runaway medical costs have made health insurance a significant challenge and margins also have been by changes in the mix of life-insurance products consumers now demand - the business men 's assurance unit represented about $ N million of the company 's $ N million in N revenue and the unit 's operating income was about $ N million said the spokesman - 's investment banker alex brown & sons inc. has been authorized to contact possible buyers for the unit - transportation ltd. said it raised its stake in ltd. of to N N from N N - a spokesman for declined to disclose the price the toronto transportation and waste services concern paid for the additional shares which he said were acquired over the last couple of weeks - the spokesman said would n't increase its stake in beyond N N without a great deal of thought because of british takeover regulations that require a company acquiring more than N N to extend an offer to the rest of the company 's shareholders - a security services and auctions company trades on london 's stock exchange - is by canadian pacific ltd. a montreal transportation resources and industrial holding concern - co. a japanese maker of video games electronic information systems and playing cards posted a N N unconsolidated surge in pretax profit to N billion yen $ N million from N billion yen $ N million for the fiscal year ended aug. N - sales surged N N to N billion yen from N billion - net income rose N N to N billion yen from N billion - net fell to N yen from N yen because of expenses and capital adjustments - without detailing specific product credited its bullish in sales including advanced computer games and television entertainment systems to surging sales in foreign markets - export sales for leisure items alone for instance totaled N billion yen in the N months up from N billion in the previous fiscal year - domestic leisure sales however were lower - hertz corp. of park n.j. said it retained merrill lynch capital markets to sell its hertz equipment rental corp. unit - there is no pressing need to sell the unit but we are doing it so we can concentrate on our core business automobiles in the u.s. and abroad said william hertz 's executive vice president - we are only going to sell at the right price - hertz equipment had operating profit before depreciation of $ N million on revenue of $ N million in N - the closely held hertz corp. had annual revenue of close to $ N billion in N of which $ N billion was contributed by its hertz rent a car operations world-wide - hertz equipment is a major supplier of rental equipment in the u.s. france spain and the - it supplies commercial and industrial equipment including and electrical equipment and trucks - inc. reported a net loss of $ N million for the fiscal third quarter ended aug. N - it said the loss resulted from and introduction costs related to a new medical equipment system - in the year-earlier quarter the company reported net income of $ N or N cents a share - the manufacturer of diagnostic systems based in pa. reported a nine-month net loss of $ N million compared with net income of $ N million or N cents a share for the nine-month period a year earlier - in over-the-counter trading fell N cents to $ N - corp. expects to report third-quarter net of about $ N million or $ N a share down from $ N million or $ N a share a year earlier richard p. simmons chairman and chief executive officer told institutional investors in new york - sales for the producer of specialty and other materials fell to about $ N million in the third quarter from $ N million a year earlier he said - he said the third-quarter estimate indicates profit for the nine months of $ N a share almost equal to the full-year N earnings of $ N million or $ N a share - in the first nine months of N net was $ N million or $ N a share - mr. simmons said the third-quarter results reflect continued improvements in productivity and operating margins - he said capital spending next year will rise to about $ N million from about $ N million this year - u.s. banknote co. said it again extended the expiration date of its $ tender offer for international banknote co. to nov. N - u.s. banknote said it is in negotiations to sell certain facilities which it did n't name to a third party and it needs the extension to try to reach a definitive agreement on the sale - u.s. banknote said it believes the sale if completed apparently would satisfy antitrust issues raised by the u.s. justice department about u.s. banknote 's offer to buy international banknote - both of the new york-based companies print stock certificates and currency - u.s. banknote said there can be no assurance a sale agreement would be concluded - it also said the tender offer would probably have to be extended further to complete financing arrangements - u.s. banknote said citibank extended the expiration date of its commitment for senior secured financing to nov. N - the offer made june N has been extended several times - closely held u.s. banknote offered the $ N a share or $ N million for as many as N million shares or N N of international banknote 's shares outstanding - u.s. banknote said that as of oct. N N million shares or about N N of the fully diluted shares outstanding had been tendered - gitano group inc. said it agreed to buy N N of regatta sport ltd. a closely held apparel maker with the assumption of $ N million of contingent debt - under the terms of the contract new york-based gitano has the option to acquire the remaining N N of regatta a maker of men 's and women 's clothes sold primarily in department stores under certain conditions - that N N is now held by clifford parker regatta 's president and chief executive officer who will continue to manage regatta 's operations under gitano - in N regatta will have sales in excess of $ N million and will show a profit mr. parker said - gitano which makes apparel sold mainly through mass like k mart and said the regatta acquisition will enhance its strategy to expand into department stores - this fall gitano began manufacturing moderately priced clothes aimed at department stores under the trademark which gitano recently acquired - enron corp. houston said the sale of preference units of its newly formed enron partners l.p. master limited partnership subsidiary will result in an gain in the fourth quarter - in the year-ago quarter the natural gas concern had net income of $ N million or N cents a share on revenue of about $ N billion - those results included a $ N million charge related to the retirement of debt - in a related move enron said it increased the number of the partnership 's units it will offer to N from N - the old and revised numbers both include provisions - enron said each unit will be priced in the $ range and will represent about N N of the partnership equity - net proceeds from the offering are expected to be close to $ N million - goldman sachs & co. and drexel burnham lambert inc. are lead underwriters - arthur m. goldberg said he extended his unsolicited tender offer of $ N a share tender offer or $ N million for di giorgio corp. to nov. N - dig acquisition corp. the new jersey investor 's acquisition vehicle said that as of the close of business yesterday N shares had been tendered - including the stake dig already held dig holds a total of about N N of di giorgio 's shares on a fully diluted basis - the offer which also includes common and preferred stock purchase rights was to expire last night at midnight - the new expiration date is the date on which dig 's financing commitments which total about $ N million are to expire - dig is a unit of dig holding corp. a unit of rose partners - mr. goldberg is the sole general partner in rose partners - in august di giorgio a san francisco food products and building materials marketing and distribution company rejected mr. goldberg 's offer as inadequate - in new york stock exchange composite trading yesterday di giorgio closed at $ N a share down $ N - what does n't belong here - a. b. black-and-white c. radio shows - if you black-and-white you 're right - after years of into the background photography is coming back - trendy magazine advertisements feature stark black-and-white photos of hollywood pitching jeans shoes and liquor - portrait studios accustomed to shooting only in color report a rush to black-and-white portrait orders - and black-and-white photography classes are crowded with students - what 's happening in photography the popularity of black and white in fashion home and - on seventh avenue designers have been advancing the look with clothing done entirely in black and white - and classic black-and-white movies are enjoying a comeback on videocassette tapes spurred in part by the backlash against of old films - the is back to black and white says richard the general manager of eastman kodak co. 's professional photography division - until two years ago sales of black-and-white film had been declining steadily since the 1960s - but last year buoyed by increased use in advertising and other commercial applications sales increased N N and they are expected to jump at least that much again this year - photographic companies are scrambling to tap the market some black-and-white product lines and developing new ones - at kodak which largely ignored the market for years black-and-white film sales now account for nearly N N of the company 's $ N billion in film and paper sales annually up from N N three years ago - the rochester n.y. photographic giant recently began marketing N one of the fastest and most sensitive films - aimed at commercial the film can be used in very low light without quality says donald of newsletter - also trying to a portion of the $ N industry is corp. a unit of ag - recently signed olympic gold to a new line of black-and-white paper that 's geared to consumers and will compete directly with kodak 's papers - slated for market by the end of the year the paper could have been introduced a long time ago but the market was n't there then says an spokesman - the biggest of the black-and-white revival is likely to be international paper co. 's division known in the industry for its premium products - sales of 's four of black-and-white film this year are growth in the overall market although the company wo n't say by exactly how much - we hope the trend lasts says 's marketing communications director - why all the interest - for baby boomers who grew up being in color black and white seems and exotic - it has an almost quality to it says owen b. butler the chairman of the applied photography department at rochester institute of technology - you can shift out of reality with black and white he adds - such features have been especially attractive to professional and marketing executives who have been steadily increasing their use of black and white in advertising - processing of black-and-white commercial film jumped N N last year to N million rolls - consider gap inc. whose latest ad campaign features black-and-white shots of hollywood stars artists and other well-known the retailer 's jeans and - richard the account manager for the campaign says gap did n't intentionally choose black and white to its ads from the color spreads of competitors - we wanted to highlight the individual not the environment he says and black and white allows you to do that better than color - the campaign won a award as this year 's best ad by a specialty retailer - even food products and automobiles which have long depended on color are making the switch - companies feel black and white will convey a stronger statement says marc l. a chicago who is working on a black-and-white print ad for food corp. 's lean - other companies that are currently using ads include american express co. and america inc - portrait studios have also onto the trend - using black and white we can make look like stars says john - his photography studio in ore. doubled its business last year and he says is booked solid for the next five - one customer says she a color portrait for black and white because it 's more dramatic - i show it to my friends and they all say - it is n't ordinary like color - still most consumers are n't black-and-white film into their cameras to take family - one big obstacle is that few develop the film anymore - typically it must be to a handful of processors and may take a week or more to be processed and returned - black-and-white film costs consumers a little less than color film and processing costs the same - but for developing costs for black-and-white film are higher - some companies are starting to tackle that problem - for example recently introduced a black-and-white film that can be processed quickly by color labs - intent on wooing customers the company is also increasing its of black-and-white photography classes - similarly is scores of photography at high schools and colleges offering free black-and-white film and paper as prizes - and kodak is distributing an video to processors on how to develop its film more efficiently - other companies are introducing related products - charles co. a leading maker of photographic introduced last month a complete targeted at who want to process their own black-and-white photographs - the which has a suggested retail price of $ N and has already become a was introduced after retailers noticed numerous requests from parents for children 's photography equipment - it seems computers as have says ian 's chairman and chief executive officer - but some industry observers believe the of black and white is only a fad - they cite the emergence of still electronic photography more newspapers turning to color on their pages and improvements in the quality of color prints - black and white has n't made the same quantum in technological development as color says mr. butler of the rochester institute - the color print today is far superior to prints of N years ago - you ca n't say the same with black and white - but when popular photography a leading magazine for selected N of the greatest photos ever made for its latest issue celebrating photography 's anniversary all were black and white - it 's got a classic spirit and carries over says alfred of professional of america - that 's the appeal - newspapers inc. said improvements in advertising and subscription revenue led to a N N gain in third-quarter profit to $ N million or N cents a share from $ N million or N cents a share - sales rose more than N N to $ N million from $ N million - the sacramento calif. company also attributed improved performance to a lower effective tax rate and higher interest income - for the nine months the newspaper chain had almost a N N increase in profit to $ N million or N cents a share from $ N million or N cents a share - sales grew almost N N to $ N million from $ N million - publishes the sacramento calif and wash news tribune and other papers in western states - in composite trading on the new york stock exchange the company closed at $ N a share down N cents - agip s.p a. and societe national the state oil companies of italy and france respectively submitted an offer to buy suisse s.a - the price was n't disclosed - a spokesman for said that the swiss oil concern was the offer submitted last friday along with two other offers also submitted last week - those two offers were private and the spokesman refused to identify the bidding companies - the spokesman further said that at least two more offers are expected from other companies within two weeks - suisse owns an oil refinery in switzerland with a capacity of N barrels a day along with a network of gasoline retailing outlets - while friday 's plunging stock market prompted new fears about the economy 's prospects a indicator that has the economy 's ups and by exceptionally long lead times points to a sustained rise in overall business activity - the barometer developed by analysts at columbia university 's center for international business cycle research here reached a record high of N in august the latest month available and the columbia researchers estimate that it has moved even higher since then - the latest reading of N was up from N in july and N as recently as march - the august rise marked the fifth straight monthly gain for the indicator which uses the N average as a base of N - in contrast the commerce department 's widely followed index of leading indicators while up in august has fallen repeatedly since reaching a high early this year - its behavior through much of N has prompted some to anticipate the start of a new recession perhaps before year 's end - but the far stronger showing of the columbia index makes a recession any time soon highly unlikely says h. moore the director of the columbia facility - a leading authority on the business cycle mr. moore also is a member of the business cycle dating group the panel of private economists that decides for the government when and recessions begin and end - the group normally only when a change in the economy 's general course seems likely - no meeting is scheduled because the expansion shows no sign of going off the tracks mr. moore reports - based largely on the recent strength in their index called the long leading indicator the columbia analysts economic growth through the rest of this year and next year as well - they expect a N N rise in N in the gross national product after adjustment for inflation - underlying this optimism is the index 's longstanding ability to signal recessions or as the case may be by substantially greater periods than the commerce department 's index of leading indicators - over the full war ii era the columbia index on the average has entered sustained declines N months before the of recessions and turned up eight months before - the comparable lead times for the commerce index whose components include the stock market are far shorter N months before recessions and only three months before - the columbia economists also have how the long leading index would have behaved had it existed in N before the stock market crash in october that in the great depression - the indicator reached a peak in january N and then fell steadily up to and through the crash - it was an entirely different pattern from what we 're seeing now mr. moore says - a major source of the recent strength in the long leading indicator has been the performance of the dow jones corporate index which is not a part of the commerce index - in august the bond measure was at its highest monthly average since early N - it also rose last friday while the stock market sagged - other components of the long leading indicator include a ratio of prices to unit labor costs in manufacturing industries the version of the money supply adjusted for inflation and the volume of new permits - notably from the columbia index is the stock market which mr. moore says is simply no longer such a good indicator of the economy 's prospects though it still is useful for anticipating some and turns - as recently as N the stock market as reflected in the standard & poor 's index of N common stocks was rated by the national bureau of economic research as the best of the N leading indicators that then made up the commerce index - it was assigned a mark of N out of a possible N compared with scores ranging as low as N for the other components - the stock market has lost some power analysts at the columbia center claim because of the growing impact of international developments - stocks have become more sensitive to factors not directly tied to the domestic economy mr. moore says citing the exchange rate for the dollar on currency markets the balance and inflows of foreign capital - he also feels that the rise of such practices as program trading has diminished the stock market 's to the economic outlook - bsn s.a. a leading french food group said it agreed to acquire g.m.b h. a west german pasta maker - the value of the acquisition was n't disclosed - the move is in line with bsn 's strategy of gradually building its share of the european pasta market through external growth - bsn will initially acquire a N N interest in a closely held concern - the french group has an agreement giving it the right to buy all the shares outstanding and this could be completed within a few months a bsn spokeswoman said - the takeover was submitted for approval by the west german office bsn said - is west germany 's producer of pasta with sales of N million marks $ N million in N - it has N workers at three production units in southwest germany and is that nation 's leading producer of pasta - the acquisition bsn 's position in the european pasta market - the french group currently ranks second after group of italy whose sales are in the italian market - moody 's investors service inc. said it reduced its rating on $ N million of senior and subordinated debt of this thrift holding company to c from ca saying it believes bondholders will recover only negligible principal - the agency said it confirmed american continental 's preferred stock rating at c - american continental 's thrift unit los angeles-based lincoln savings & loan association is in and the parent company has filed for protection from creditor lawsuits under chapter N of the federal bankruptcy code - centrust savings bank miami - moody 's investors service inc. downgraded its ratings on the subordinated debt of centrust to from - the rating agency also reduced the ratings for long-term deposits to from and for preferred stock to ca from - the rating agency said about $ N million in securities was affected - the were prompted moody 's said by the continuing turmoil in the junk bond market and the suspension of dividends on centrust 's preferred stock - moody 's also said it believed the proposed sale of N centrust branches to great western bank could if completed endanger the thrift 's funding and market position - the stock market avoided a repeat of black monday as prices recovered from an early slide spurred by bargain-hunting institutions and program traders - the dow jones industrials closed up N points at N the gain ever after being down as much as N points in the morning - the rally erased about half of friday 's 190.58-point plunge but analysts are cautious about the market 's outlook - the dollar also rebounded while bond prices plummeted and treasury bill rates soared - junk bonds also recovered somewhat though trading remained stalled - gold also rose - tokyo stock prices bounced back in early trading tuesday following a N N plunge on monday - the dollar also moved higher in tokyo - donald trump withdrew his $ N billion offer for american air citing the recent change in market conditions - amr slid $ N to $ N - also a ual group tried to get financing for a lower bid possibly $ N a share - ual fell $ N to $ N - leveraged buy-outs of airlines would be subject to approval by the transportation secretary under a bill passed by a house subcommittee - ibm 's earnings tumbled N N in the third quarter slightly more than expected - the computer giant partly cited a stronger dollar and a delay in shipping a new high-end disk drive - analysts are about ibm 's outlook for the next few quarters - u.s. auto makers plan to decrease car production N N in the fourth quarter with virtually all the decline coming from the big three - output at and managed plants in the u.s. is due to rise N N - budget director darman said he wo n't give federal agencies much in coping with gramm-rudman spending cuts which took effect yesterday - darman hopes to congress to finish a deficit plan - the s&l bailout agency would be restricted by a new bill in how it raises capital - the ways and means plan would create another possible obstacle to selling sick thrifts - a natural gas rule was struck down by a federal appeals court - the regulation had prevented pipeline firms from passing part of $ N billion in costs along to customers - the supreme court agreed to decide whether a federal court may a merger that has won regulatory approval but been ruled in a private suit - merrill lynch 's profit slid N N in the third quarter - bear stearns posted a N N gain while painewebber had a decline due to a year-ago gain - blue arrow of britain plans to return to the name manpower and take a big write-off - the moves may help the firm its dominance of the u.s. market - j.p. morgan posted a $ N billion loss for the third quarter reflecting a big addition to loan-loss reserves - ncnb 's profit more than doubled - k mart agreed to acquire pace membership warehouse for $ N million expanding its presence in the growing business - markets - stocks volume N shares - dow jones industrials N up N transportation N off N utilities N up N - bonds shearson lehman hutton treasury index N off - commodities dow jones futures index N off N spot index N up N - dollar N yen off N N marks off N - monday october N N - the key u.s. and foreign annual interest rates below are a guide to general levels but do n't always represent actual transactions - prime rate N N N - the base rate on corporate loans at large u.s. money center commercial banks - federal funds N N N high N N N low N N N near closing bid N N N offered - reserves traded among commercial banks for overnight use in amounts of $ N million or more - source fulton prebon u.s.a inc - discount rate N N - the charge on loans to depository institutions by the new york federal reserve bank - call money N N N to N N - the charge on loans to brokers on stock exchange collateral - commercial paper placed directly by general motors acceptance corp. N N N to N days N N N to N days N N N to N days N N N to N days N N N to N days N N N to N days N N N to N days - commercial paper high-grade unsecured notes sold through dealers by major corporations in multiples of $ N N N N days N N N days N N N days - certificates of deposit N N one month N N two months N N three months N N six months N N one year - average of top rates paid by major new york banks on primary new issues of negotiable c.d.s usually on amounts of $ N million and more - the minimum unit is $ N - typical rates in the secondary market N N one month N N three months N N six months - bankers acceptances N N N days N N N days N N N days N N N days N N N days N N N days - negotiable bank-backed business credit instruments typically financing an import order - london late eurodollars N N N to N N N one month N N N to N N N two months N N N to N N N three months N N N to N N N four months N N N to N N N five months N N N to N N N six months - london interbank offered rates libor N N N one month N N N three months N N N six months N N N one year - the average of interbank offered rates for dollar deposits in the london market based on quotations at five major banks - foreign prime rates canada N N germany N N japan N N switzerland N N britain N N - these rate indications are n't directly comparable lending practices vary widely by location - treasury bills results of the monday october N N auction of short-term u.s. government bills sold at a discount from face value in units of $ N to $ N million N N N weeks N N N weeks - federal home loan mortgage corp freddie mac posted yields on 30-year mortgage commitments for delivery within N - N N standard conventional mortgages N N N N rate capped one-year adjustable rate mortgages - source telerate systems inc - federal national mortgage association fannie mae posted yields on N year mortgage commitments for delivery within N days priced at par N N standard conventional fixed N N N rate capped one-year adjustable rate mortgages - source telerate systems inc - merrill lynch ready assets trust N N - annualized average rate of return after expenses for the past N days not a forecast of future returns - intel corp. said it reached an agreement with computer systems corp. to develop software standards for intel 's microprocessor - the introduced earlier this year is intel 's entry in the crowded market for reduced instruction set computing or risc computers - intel based in santa clara calif. is the market leader for traditional microprocessors with its N family that forms the heart of personal computers - under the agreement intel will invest $ N million to acquire a N N stake in a maker of for scientists and engineers - based in mass. will license its technologies to intel providing users a way to let many microprocessors in a single computer work on a problem simultaneously - said it plans to use the microprocessor in future products - it declined to discuss its plans for upgrading its current product line - inc. which intends to expand its position in the medical and markets said it acquired a cotton and products division from closely held products corp. for $ N million - said it expects the division to add substantial sales volume and to make a positive contribution to our earnings in N and beyond - in N the cincinnati company earned $ N million or N cents a share on revenue of $ N million - said the division operates under the trade name and supplies the medical and markets - the business based in st. louis had sales of more than $ N million in the fiscal year ended march N said - burmah oil plc a british independent oil and specialty chemicals marketing concern said shv holdings n.v. of the netherlands has built up a N N stake in the company - james alexander a burmah spokesman said shv had previously owned a little under N N of burmah for about two years - the dutch company had n't notified burmah of its reason for increasing the stake he said - shv which last year merged its north sea oil and gas operations with those of group plc has been pegged by speculators as a possible suitor for burmah oil in recent weeks - shv also owns N N of - burmah which owns the brand of oils reported a N N rise in net income to # N million $ N million in the first half - j.p. industries inc. said it signed a definitive agreement to sell its builders ' hardware group to closely held inc. of beverly hills calif - terms were n't disclosed but a j.p. industries spokesman said the amount j.p. industries will get for the group is a little better than expected by the marketplace and the marketplace had been expecting $ N million to $ N million - the group consists of corp. and modern inc - j.p. industries which is based in ann mich. said the sale a previously announced program to itself of its hardware and plumbing supplies operations - the company 's remaining business is the manufacture and sale of engine and products for industrial and transportation applications - citing a $ N million provision for doubtful accounts dallas-based national heritage inc. posted a loss for its fourth quarter ended june N - a unit of troubled southmark corp. the operator of nursing homes and retirement centers said it sustained a net loss of $ N million or nine cents a share compared with net income of $ N million or eight cents a share a year earlier - operating revenue rose N N to $ N million from $ N million in the year-earlier quarter - the company said the $ N million reserve was created to reflect doubt about the of receivables owed to national heritage by some of the real estate partnerships it manages - the company also said expenses incurred by the previous board and management in the recent contest for control were recognized primarily in the first quarter ended sept. N - national heritage stock fell N cents yesterday to close at $ N a share in new york stock exchange composite trading - united biscuits holdings plc a british food producer announced the creation of a european group to bring together its trading interests in the region - the new group will all of united 's manufacturing and marketing operations in the food sector apart from those based in the u.s. - united biscuits said the combined group which will include businesses such as biscuits and terry 's will have annual sales of more than # N billion $ N billion and trading profit of more than # N million $ N million - the new structure will enable united biscuits to focus clearly upon opportunities for planned growth during the 1990s said bob deputy chairman and group chief executive - last month united biscuits agreed to sell its entire restaurant operations to grand metropolitan plc for # N million - an american journalist now is standing trial in namibia - this is the place that world opinion has been celebrating over in the expectation that it 's going to hold an election - the most likely winner will be the swapo rebels - the u.s. journalist 's crime was writing that the head of the commission charged with overseeing the election 's fairness was openly sympathetic to swapo - shortly after that mr. had scott stanley arrested and his confiscated - mr. stanley is on trial over charges that he violated a issued by the south african administrator general earlier this year which made it a crime punishable by two years in prison for any person to or the election commission - the stanley affair does n't well for the future of democracy or freedom of anything in namibia when swapo starts running the government - to the extent mr. stanley has done anything wrong it may be that he is out of step with the consensus of world intellectuals that the guerrillas were above all else the victims of by neighboring south africa - swapo has enjoyed favorable western media treatment ever since the u.n. general assembly declared it the sole representative of namibia 's people in - last year the u.s. a peace settlement to remove cuba 's from and hold free and fair elections that would end south africa 's control of namibia - the elections are set for nov. N - in july mr. stanley editor of american press international a washington conservative wire service visited namibia to report on the election campaign - he interviewed mr. head of a commission charged with investigating electoral and reported that mr. was openly sympathetic to swapo and indeed had defended its leaders in court - after mr. stanley 's article was published in two newspapers mr. had criminal charges brought against their editors publisher and lawyer - mr. stanley was arrested and charged along with the others when he returned to namibia this month - both the state department and the lawyers committee for freedom of the press have mr. stanley 's - mr. stanley 's arrest is the latest in a series of incidents that threaten to namibia 's elections - both south african and swapo are voters - the u.s. is in the habit of arranging peace settlements and then its hands over the results - it now has the chance to that record in namibia - state and the human-rights community should insist that mr. stanley and his fellow defendants be released and that the united nation 's monitors make certain that mr. commission election complaints from all sides - commodity futures prices generally reflected the stability of the stock market following its plunge friday - yesterday the stock market 's influence at first created nervousness - later however it became more of an than a force as individual markets reacted more to their own factors - gold the traditional haven in times of financial crisis continued its with the dollar rising early in the day as the currency fell and then giving up some of its gains as the dollar recovered - copper and crude oil reacted sharply to the concern that a crash yesterday could have a potentially devastating effect on the economy - copper fell and showed little rebound through the day as one of the major supply problems that had been supporting prices appeared to be solved - crude oil declined early but as the stock market retained early gains it too became stronger ending with a small net loss - trading in cotton and sugar was nervous and showed small declines - in chicago grain and soybean prices rose slightly - livestock and meat prices however dropped on concern that a financial crisis would cut consumption of beef and pork - in commodity markets yesterday precious metals futures prices were moderately higher as gold gave up some of its early gains and platinum behaved first falling and then later rising - silver performed quietly - the spot october gold price rose $ N to $ N an ounce - the more active december delivery gold settled with a gain of $ N an ounce at $ N after trading as high as $ N - december silver was up N cents an ounce at $ N - platinum behaved more like an industrial metal easing early on concern over a possible weaker economy but recovering later as the stock market strengthened - gold was nowhere the spectacular performer it was two years ago on black monday - for one thing last friday precious metals markets closed before the stock market went into its nose dive so it could n't react to it - back on friday oct. N the stock market declined during the day and gold prices surged as the stock market fell - the october N contract that day rose as much as $ N to as high as $ N and the more deferred positions due to mature as late as march N rose as much as $ N - on black monday oct. N N the october contract tacked on further gains rising to as high as $ N for a gain of almost $ N on top of the friday advances before giving up almost $ N of that at the close - yesterday 's october gain of $ N was compared with that - one analyst peter of & co. new york said the gold market already had some good technical factors that would have caused prices to rise with or without the stock market - what the stock market did was cause the rise to take place earlier than it would have happened said mr. - there 's a good chance that gold will retain its gains and rise further he said - he expects a drop in interest rates which would help gold by keeping the dollar from rising - finally according to mr. the impact of the strong dollar should be reflected in reduced exports in the august merchandise trade deficit when the figures are released today - this would be damaging to the dollar and supportive for gold he said - energy - worried that friday 's 190-point stock market plunge might be a of things to come for the economy petroleum futures traders called a halt to the recent string of increases in crude oil futures prices - the u.s. benchmark crude west texas intermediate closed at $ N a barrel for november delivery down N cents - some analysts said crude was due for a correction following several days of significant gains - but most market observers agreed that friday 's stock market drop is what spirits in the petroleum pits yesterday - until yesterday futures prices had been headed up on expectations that world oil demand will continue to be strong - the organization of petroleum exporting countries increased its production ceiling for the fourth quarter based on projections of robust demand - so any bearish indicator such as friday 's drop in the stock market sends through the oil markets as well - indeed after reacting early in the trading day to friday 's plummet futures prices firmed up again as traders took note of the stock market 's partial recovery yesterday - copper - futures prices fell and showed little rebound as one major labor problem that had been prices appeared to be solved - the december contract declined N cents a pound to $ N - prices were down from the outset of trading on concern that a drop in the stock market might create a weakened economy and a reduction in copper use - but the recovery in the stock market provided little help for copper as word spread that a three-month strike at the highland valley mine in british columbia was about over according to an analyst - highland valley is a large canadian producer and principal supplier to japan which recently began seeking copper elsewhere as its inventories shrank - last week it was reported that company and union negotiations had overcome the major hurdle the contracting out of work by the company - now the analyst said only minor points remain to be up - for all and purposes an agreement appears to have been achieved he said - copper inventories in new york 's commodity exchange warehouses rose yesterday by N tons to N tons - london metal exchange copper inventories last week declined N tons to N tons - the stocks decline was about as expected but the comex gain was n't - however this was brushed aside by concern over the stock market the analyst said - at one point in futures trading as the stock market firmed the december contract rose to as high as $ N but it was n't able to sustain the gain - it was simply he said and selling by funds that are computer helped depress prices - cotton - futures prices eased more in reaction to hurricane jerry than to any influence of the stock market - the december contract ended with a loss of N cent a pound at N cents - technical considerations following the hurricane which was a factor in the market friday caused prices to decline yesterday said ernest simon cotton specialist for prudential-bache securities new york - prices rose sharply friday as the storm approached texas and louisiana which is part of the mississippi delta area - however after the potential effect of the hurricane prices began to slip late friday mr. simon said - that selling continued yesterday and kept prices under pressure he said - weather is being predicted for the high plains of texas and the northern states of the delta during the coming weekend mr. simon said - that has n't yet captured traders ' attention he added - sugar - futures prices declined - the march contract was off N cent a pound at N cents - at one point in early trading the march price rose to as high as N cents when the stock market recovered but the price then fell back - a factor one analyst said was that india which had been expected to buy around N tons of sugar in the world market did n't make any purchases - india recently bought N tons and was expected to buy more the analyst said - another analyst thought that india may have pulled back because of the concern over the stock market - india may have felt that if there was a severe drop in the stock market and it affected sugar it could buy at lower prices said analyst for shearson lehman hutton new york - at any rate she added india needs the sugar so it will be in sooner or later to buy it - farm products - the prices of cattle and futures contracts dropped sharply because traders speculated that the stock market plunge friday will in the minds of u.s. consumers long enough to prompt them to rein in their spending at the supermarket which would hurt demand for beef and pork - the price of the contract for october delivery dropped its maximum permissible daily limit of N cents a pound - the prices of most grain futures contracts rose slightly yesterday out of relief that the stock market was showing signs of recovering - earlier in the session the prices of several soybean contracts set new lows - a broad rally began when several major processors began buying futures contracts apparently to take advantage of the price dip - knight-ridder inc. said it would report increased earnings per share for the third quarter contrary to reported analysts ' comments that the publishing company 's earnings would be down - a company spokesman said he believed the confusion was caused when james knight-ridder 's chairman and chief executive told new york analysts two weeks ago that knight-ridder 's earnings per share for the first nine months of N would be behind a little bit from like period of - the knight-ridder spokesman said the third-quarter earnings that the company plans to report oct. N are expected to be up - the spokesman said he was comfortable with revised analysts ' projections that the company would report earnings of between N cents and N cents a share compared with the N cents a share it reported for the N third quarter - knight-ridder said it agreed with estimates that net income for all of N would be around $ N a share compared with $ N a share a year earlier - in new york stock exchange composite trading yesterday knight-ridder closed at $ N down N cents - dd acquisition corp. said it extended its $ offer for dunkin donuts inc. to nov. N from yesterday - the offer has an indicated value of $ N million - dd acquisition is a partnership of unicorp canada corp. 's capital group unit and cara operations ltd - as previously reported under the terms of a agreement with dunkin donuts the partners agreed to keep their offer open until nov. N and not to acquire any additional shares except through a tender offer on that date - dd acquisition said that it already owns N N of the common shares of the shop chain and that as of the close of business friday an additional N N had been tendered to its offer - dunkin donuts is based in mass - cara operations a food services concern and unicorp a holding company with interests in oil and natural gas and financial services are based in toronto - golden west financial corp. riding above the turbulence that has troubled most of the thrift industry posted a N N increase of third-quarter earnings to $ N or N cents a share - the company earned $ N million or N cents a share in the year-ago quarter - herbert m. chairman and chief executive officer of the oakland calif. savings-and-loan holding company credited the high number of loans added to the company 's portfolio over the last N months for its earning asset base and improving profit performance - however the executive noted that demand for new mortgages depressed new loan to $ N billion N N below the same period last year - in savings activity mr. said consumer deposits have enjoyed a steady increase throughout N and topped $ N billion at quarter 's end for the first time in the company 's history - deposit growth amounted to $ N million more than double the year-ago figure - corp. benton harbor mich. said it has developed a process to recover environmentally harmful chlorofluorocarbons or cfcs that previously entered the atmosphere during repair of refrigerators and - the maker of home appliances said the process which involves the use of a plastic bag during repairs to capture the substance and transport it to a recycling center is already in use at a number of its service centers and will be available to all authorized repair centers by spring - earlier repairs the cfcs out of the home through a directly into the atmosphere - cfcs are widely used as and fire - but their use has been linked to a potentially dangerous depletion of the earth 's ozone layer and a number of companies are seeking to curtail use or at least of the substance - said we see this process as a small but important step toward eventual elimination of use in manufacture - energy corp. dallas said it discovered a new oil field northeast of its previously discovered field in the southeast area of indonesia - said it did n't run a production test on the three discovery wells it in the field which is about N miles from the field because the wells are similar to others at its and fields - however said it believes the reserves in the field are about N million barrels of oil - the field has estimated reserves of N million barrels and the field has estimated reserves of N million barrels - an independent oil and gas concern is the operator and owns a N N interest in the new field called northeast - other interests are owned by petroleum development ltd. c. energy co. ltd. g.m.b h. production ltd. oil indonesia ltd. co. ltd. ltd. shell a.g. and oil co - the contract area is held with the state oil company - environmental systems co. said it is its results to reduce its reported net income for the first nine months of its fiscal year after it took tax credits that already had been taken last year - the little rock services company said the will reduce its net for the nine months ended july N to $ N million or N cents a share from $ N million or N cents a share - net for the third quarter restated is $ N million or N cents a share - the company previously reported net of $ N million or N cents a share - the company said that for financial reporting purposes last year it took tax credits that will be recognized for tax purposes this year - but because of confusion it took those credits again in reporting its results through the first nine months - jack w. environmental systems president and chief executive officer said the change increases the company 's effective tax rate to about N N from N N - memotec data inc. said it signed a definitive merger agreement with isi systems inc. under which memotec will acquire isi for $ N u.s. a share or about $ N million in cash and securities - in american stock exchange composite trading isi closed up $ N at $ N - in montreal exchange trading memotec closed unchanged at N canadian dollars us$ N - memotec said under the agreement isi a mass. provider of computer software and services to the insurance industry will merge with a u.s. unit of memotec created for that purpose - memotec is a maker of telecommunications products and provider of telecommunications and computer services - memotec said the agreement calls for it to make a $ cash tender offer for all shares outstanding of isi - but it said charles johnston isi chairman and president agreed to sell his N N stake in isi to memotec upon completion of the tender offer for a combination of cash memotec stock and debentures - memotec said the tender offer is on among other things holders at least N N of the shares outstanding other than the shares held by mr. johnston - isi said its board has instructed management to accept inquiries from any others interested in making a bid - isi said it can withdraw from the merger agreement with memotec if a better bid - cms energy corp. jackson mich. said it has resumed the purchase of its common stock under a program approved by its directors in N - at the time of the original announcement cms said its board authorized the purchase of as many as five million of its shares - a spokesman said N million shares have been purchased since then - the company said it will buy additional shares from time to time in the open market or in private transactions at prevailing market prices - in composite trading on the new york stock exchange cms energy closed at $ N a share down N cents from the closing price of $ N a share on thursday before friday 's plunge - the utility company currently has about N million shares outstanding - morgan stanley & co. will act as the exclusive broker for the repurchase - hughes aircraft co. a unit of general motors corp. said it agreed to purchase the technology division of corp - terms of the agreement were n't disclosed - but for the fiscal year ended july N N the most recent period for which results were broken out the unit accounted for more than half the $ N million in sales recorded by the company 's government systems sector - which is based in conn. said the sale of the conn. unit is consistent with its restructuring strategy announced in april - in addition to making systems the unit also makes laser warning - these are used aboard military to warn pilots that a laser weapon has been focused on them - hughes of los angeles said the unit 's work efforts by its and data systems group which makes military and night vision equipment - hughes said it expects the sale to close by year end - the communications workers of america ratified a new regional contract and all but one of the local agreements with bell atlantic corp - 's new jersey commercial local which represents about N service representatives and marketing employees rejected the tentative agreement - both the union and the regional telephone company said they were working together to resolve differences - the new three-year contracts which replace ones that expired aug. N cover N bell atlantic employees - the follows a strike against the company - meanwhile and international of electrical workers members remain on strike against nynex corp. the new york-based regional phone company - the unions and the company last week agreed to - the represents N nynex workers and the represents N workers - for the moment at least euphoria has replaced anxiety on wall street - the dow jones industrial average jumped sharply yesterday to close at N panic did n't sweep the world 's markets and investors large and small seemed to accept friday 's dizzying 190-point plunge as a sharp correction not a - many went bargain-hunting - among those with relief was john h. gutfreund chairman of salomon brothers who took to the firm 's trading floor to monitor yesterday 's events - as the rally gained strength at N p.m. he broadly his and stanley his top stock trader on the back - at first it seemed as if history might repeat itself - as trading opened yesterday morning on the big board stocks of many of the nation 's biggest companies could n't open for trading because a wave of sell orders was overwhelming buyers - by N the dow industrials were off N points and the stock of ual corp. whose troubles had kicked off friday 's plunge still had n't opened - but then as quickly as the dow had fallen it began to turn around - it ended with a gain of N points - by the market 's close volume on the new york exchange totaled more than N million the fourth highest on record - the big board handled the huge volume without any obvious strain in sharp contrast to black monday of N - but the rally was largely confined to the blue-chip stocks which had been hard hit during friday 's selling frenzy - overall more big board stocks lost money than gained - and many arbitragers already reeling from friday 's collapse of the ual deal were further hurt yesterday when a proposed takeover of amr corp. the parent of american airlines collapsed - indeed the dow jones transportation average plunged N points its drop in history - world-wide trading was generally - the frankfurt stock exchange was hardest hit of the major markets with blue chips there falling N N - in london a midday rally left the market 's major index off N N and tokyo 's leading stock index fell only N N in surprisingly lackluster trading - other more traded asian markets were hit harder than tokyo 's but there were no declines - investors big and small say they learned valuable since the N crash in this age of computerized trading huge or in a few hours ' time must be expected - what 's more such short-term are and are no cause for panic selling - stephen boesel a major money manager for t. rowe price in baltimore says there was less panic than in N we had been through it once - in wis. who owns a supplier of equipment and is n't active in the stock market agrees - i look at it as a matter he says - many other factors played a part in yesterday 's comeback - the federal reserve signaled its willingness to provide liquidity the interest rate on its loans to major banks inched downward early in the day - foreign stock markets which kicked off black monday with a huge selling spree began the day off by relatively modest amounts - the dollar after falling sharply in overnight trading to N yen bounced back strongly to N thus easing fears that foreigners would unload u.s. stocks - and the widely opinion among most market experts that a crash was n't in store also helped calm investors - many major institutions for example came into work yesterday ready to buy some of the blue chips they felt had been sharply undervalued on friday - still amid all the and signs of relief over yesterday 's events some market professionals cautioned that there is nothing present in the current market system to prevent another dizzying drop such as friday 's - there is too much says money manager barry - computers have increasingly connected securities markets world-wide so that a buying or selling wave in one market is often passed around the globe - so investors everywhere nervously yesterday 's opening in tokyo where the nikkei average of N blue-chip stocks got off to a rocky start - the average plunged some N points or N N in the first N minutes of trading - but the selling wave had no conviction and the market first surged upward by N points then drifted lower closing down N - unlike two years ago most of japan 's major investors chose to sit this out - in merrill lynch & co. 's tokyo trading room some N traders and sat quietly with few orders to process - clients are all staying out of the market one merrill trader says - the relative calm in tokyo proved little comfort to markets opening up in europe - frankfurt 's opening was delayed a half hour because of a crush of sell orders - the beginning was chaotic says nigel a broker for commerzbank - in london the view from the trading floor of an american securities firm jefferies & co. also was troubling - a computer screen N blue-chip stocks colors each one red when its price is falling - the screen was a sea of red - i see concern but i do n't see panic says j. francis a new yorker who runs the office - london 's blue-chip stock index turned up just before N a.m new york time sending an encouraging message to wall street - when trading opened in new york at N a.m. edt stocks fell sharply as expected - futures markets in chicago had opened at a level suggesting the dow would fall by about N points - with sell orders up from friday about half the stocks in the dow could n't open on time - by N the industrial average had dropped N points - by N a.m. it was down N - ten minutes later the dow hit bottom down N points another N N - but shortly before then some of wall street 's sharpest traders say they a turn - the first thing that caught my eye that was encouraging was treasury bonds were off says austin george head of stock trading at t. rowe price - it meant that people were n't running to the safety of bonds - shortly after N a.m. the major market index a chicago board of trade futures contract of N stocks designed to mimic the exploded upward - stock traders were buoyed because an in the mmi had also started the recovery in stocks on the tuesday following black monday - the mmi has gone better shouted a trader in the london office of shearson lehman hutton - shearson 's london trading room went wild - traders shouted out as their reuters quotron and telerate screens posted an loss on wall street - then nine minutes later wall street suddenly rebounded to a gain on the day - rally rally rally shouted shearson 's andy rosen - this is panic buying - major blue-chip stocks like philip morris general motors and & gamble led the rally - japanese were said to be heavy buyers - german and dutch investors reportedly loaded up on kellogg co - then traders say corporations with share buy-back programs kicked into high gear triggering gains in among other issues and mcdonald 's - walt disney co. which had one of the biggest imbalances on friday and was one of seven stocks that halted trading and never reopened that day opened yesterday late at N down N - but then it suddenly burst upward N as goldman sachs & co. stepped in and bought almost every share offer traders said - by N the dow had turned up for the day prompting on trading desks and exchange floors - among big board specialists the cry was pull your offers meaning that specialists soon expected to get higher prices for their shares - it was on the upside said one big board specialist - what we had was a real old-fashioned rally - this technical strength spurred buying from wall street 's black boxes computer programs designed to trigger large stock purchases during bullish periods - typical perhaps was 's dean - mr. who manages $ N billion says we turned the trading system on and it did whatever it was to do - asked what stocks the computer bought the money manager says i do n't know - not everybody was making money - the on the chicago board options exchange the nation 's major options market was heavy after the trading in s&p N stock-index options was halted friday - many market makers in the s&p N index options contract had bullish positions friday and when the shutdown came they were frozen with huge losses - over the weekend clearing firms told the chicago market makers to get out of their positions at any cost monday morning - they were absolutely killed said one chicago-based options trader - meanwhile a test of the stock market 's rally came at about N p.m. with the dow at N up N points on the day - charles a strategist at merrill lynch says bargain hunting had explained the dow 's strength up to that point and that many market professionals were anticipating a drop in the dow - moreover the announcement that real estate and sometime raider donald trump was his offer for amr corp. might have been expected to traders - instead the rally only for about N minutes and then forward as institutions resumed buying - the market closed minutes after reaching its high for the day of - across the country many people took yesterday 's events in while remaining generally uneasy about the stock market in general - says james norman the mayor of mo. i do n't invest in stocks - i much prefer money i can put my hands on - while mayor norman found the market 's performance monday reassuring he says he remains uneasy - we have half the experts saying one thing and half the other about the course of the economy - ralph a farmer and store operator in neb. says of the last few days events if anything good comes out of this it might be that it puts some of these lbos on the - says gordon fines a money manager at financial services in minneapolis you 're on a roller and that may last - the public is still cautious - skipper 's inc. wash. said it signed a definitive merger agreement for a national pizza corp. unit to acquire the N N of skipper 's inc. it does n't own for $ N a share or about $ N million - acquisition co. a national pizza unit plans to begin a tender offer for skipper 's on friday on at least two-thirds of skipper 's shares being tendered - national pizza said the transaction will be financed under its revolving credit agreement - in national over-the-counter trading skipper 's shares rose N cents to $ N - skipper 's said the merger will help finance remodeling and future growth - skipper 's previously turned down a $ proposal from national pizza and pizza hut inc. questioned whether the purchase would violate national pizza 's franchise agreements - national pizza said it settled its dispute with pizza hut allowing it to make the purchase - also skipper 's results began to turn around permitting a higher offer national pizza said - for the N weeks ended sept. N skipper 's had net income of $ N or N cents a share compared with a net loss a year earlier - revenue was $ N million - east germans rallied as officials reportedly sought honecker 's - in what was considered the largest protest in the communist state 's history at least N demonstrators marched through the southern city of leipzig to press demands for democratic freedoms opposition activists said - police did n't intervene - meanwhile as the first of more than N east germans trying to to the west through poland their a west german newspaper reported that regional communist officials demanded the dismissal of hard-line leader honecker - secretary of state baker in a foreign policy speech called for the reunification of germany saying it was the legitimate right of the german people - gorbachev blamed the soviet union 's press for contributing to the nation 's mounting problems - at a meeting friday the kremlin leader complained about recent articles that raised the of civil unrest and accused the media of fueling panic buying of goods by publishing stories about impending shortages - house-senate conferees approved a permanent smoking ban on domestic airline routes within the continental u.s. and on flights of less than six hours to alaska and hawaii - the curbs would cover all but a small percentage of flights and represent an expansion of the current ban on flights of less than two hours - e. robert was sentenced by a u.s. judge in new york to six years in prison and fined $ N for his racketeering conviction in the wedtech scandal - an associate of general was found guilty in august of taking $ N in illegal from the defense contractor - nasa resumed the for today 's launch of the space shuttle atlantis and a federal appeals court in washington dismissed a lawsuit by anti-nuclear groups to delay the flight because the galileo space probe was aboard - the space agency said it did n't expect weather or protesters to block the - the bush administration is preparing to extend a ban on federal financing of research using tissue government sources said - a temporary prohibition was imposed in march N - while anti-abortion groups are opposed to such research scientists have said such tissue could be effective in treating - delegates from N nations endorsed a ban on world ivory trade in an attempt to rescue the endangered elephant from - five african nations however said they would continue selling the valuable - held reconciliation talks with at the egyptian resort of - it was the leader 's first trip to egypt in N years - they announced a reduction in for travel but did n't show any real signs of full diplomatic ties - the egyptian president said he would visit libya today to resume the talks - seoul and reached a tentative agreement to allow visits between families on the divided korean peninsula - such family would be the second since N - differences remained between the north and south korean governments however over conditions for the exchanges - freed black resumed political activity in south africa and vowed to fight against apartheid raising fears of a possible white backlash - the nation 's main white opposition party warned that the government 's release sunday of eight black political bringing chaos and eventual black marxist rule to the nation - the white house said bush is fully satisfied with cia director webster and the intelligence agency 's performance during the oct. N failed coup in panama - the washington post reported that unidentified senior administration officials were frustrated with webster 's activities during the and wanted him replaced - poland 's legislature approved limits on automatic wage increases without special provisions for food price rises - the vote was considered a test of the government 's resolve to proceed with a harsh program - norway 's king installed a government as 's labor regime power - the cabinet is led by prime minister jan who acknowledged a difficult situation since the coalition controls only N seats in 's legislature - el salvador 's government opened a new round of talks with the country 's leftist rebels in an effort to end a civil war - a spokesman said the guerrillas would present a cease-fire proposal during the negotiations in costa rica that includes constitutional and economic changes - the state department said there was a possibility that some nicaraguan rebels were selling their arms to guerrillas but insisted it was n't an organized effort - separately secretary of state baker complained about a u.n. aide who last week told the contras to as part of a regional peace accord - died N actor and director in los angeles of - N novelist and sunday in paris of cancer - british retail sales volume rose a provisional N N in september from august and was up N N from september N the department of trade and industry said - for the three months ended in september retail sales volume was down N N from the previous three months and up N N from a year earlier - chicago investor william agreed to sell three divisions of cluett peabody & co. for about $ N million to s.a. a closely held clothing maker based in paris - shortly after completing the $ N billion acquisition of west inc. in april mr. 's holding company inc. said it was considering the sale of cluett a leading maker and one of west 's biggest units - included in the sale are cluett units that make men 's shirts under the arrow name under the gold name and through the division - the companies said the agreement is subject to 's of financing and to regulatory and other approvals - they said the sale is expected to be concluded by the end of november - mr. said the sale of three of cluett 's four main divisions plus other anticipated west asset sales by december should bring in a total of about $ N million - he did n't elaborate on other asset sales being considered - mr. followed a similar pattern when he acquired northwest industries inc. and then sold much of its assets - but he kept fruit of the inc. the underwear maker that he still controls and serves as chairman and chief executive - cluett was an independent company until west acquired it for $ N million in cash and stock in N - in the fiscal year ended sept. N N cluett had operating profit of $ N million on sales of $ N million - sells clothes under various labels including and bill robinson for men and ralph for women - a spokesman said the company had sales of $ N million in N - in new york stock exchange composite trading west fell N cents to $ N - britain 's blue arrow plc intends to change its name to manpower plc and write off a chunk of the nearly $ N billion in good will realized in the takeover of manpower inc - blue arrow chairman mitchell fromstein said in an interview that the two steps may be a prelude to the world 's biggest group in the u.s. - mr. fromstein disclosed the planned steps expected within a few months as blue arrow posted a N N drop in its third-quarter pretax earnings - the name change and good will write-off could help blue arrow 's dominance of the u.s. market and give it a more american image as u.s. investors turn jittery about foreign stocks after friday 's market plunge - u.s. holders now own more than N N of blue arrow compared with N N last january - in the u.s. market the recognition of the manpower name is stronger than blue arrow mr. fromstein said - the moves also could shareholders ' perception of blue arrow as a company in turmoil - it further the concept that blue arrow is a thing of the past said doug arthur an analyst at kidder peabody & co. in new york - the proposed changes all make a lot of sense to me he added - in a widely publicized coup mr. fromstein ousted berry as blue arrow chief executive in january a month after mr. berry had forced mr. fromstein out as the $ N chief of manpower - mr. fromstein his control in april by taking over from mr. berry as chairman - but the blue arrow is n't over yet as the british government is investigating a disputed # N million $ N million loan which mr. fromstein has said was made under mr. berry 's direction - blue arrow was able to pull off the $ N billion takeover of manpower in N largely because different british and american accounting standards produce higher reported earnings for british companies - under british rules blue arrow was able to write off at once the $ N billion in good will arising from the purchase - as a company blue arrow would have to the good will over as many as N years creating a continuing drag on reported earnings - good will is the excess of cost of an acquired firm operating unit or assets over the current or fair market value of those assets - but with so many shares now held in the u.s. blue arrow reports its earnings two ways based on both u.k. and u.s. accounting standards - our balance sheets look like they came from alice 's mr. fromstein said - the british version shows a handful of pounds of net worth following the N write-off of good will while the american version reflects $ N billion of net worth because almost none of the good will has been written off - mr. fromstein said he hopes to some of the good will left on blue arrow 's u.s. books in one fell but would n't specify how much - people close to blue arrow suggested the write-down would represent a sizable chunk with executives claiming prior management the extent of manpower 's good will - that move along with the return to the manpower name could bolster the company 's prospects during possibly difficult times for temporary help - the number of u.s. temporary workers fell about N N in the N months ending aug. N after sliding nearly N N in july said kidder peabody 's mr. arthur - blue arrow blamed the pretax profit drop in the quarter ended july N partly on slower earnings growth of units in britain - overall pretax profit slid to # N million in the quarter from # N million a year earlier - richard g. sim the man credited with applied power inc. from an into a player in the global market for tools hopes to guide a similar turnaround at the company 's latest acquisition barry wright corp - the 45-year-old former general electric co. executive figures it will be easier this time - but analysts while the acquisition say applied 's chief executive faces a tough challenge in the two companies - barry wright acquired by applied for $ N million makes equipment and systems - the mass. company 's sales have been and its profits have dropped - last year 's earnings of $ N million including $ N from a restructuring gain were far below the year-earlier $ N million - besides spurring barry wright 's sales which were $ N million in N mr. sim must its costs and product line - the question is how long it 's going to take barry wright to make a contribution says f. john an analyst at blunt ellis in milwaukee - the answer will help determine whether applied continues to reach the ambitious goals set by mr. sim - the butler wis. manufacturer went public at $ N a share in august N and mr. sim 's goal then was a $ N per-share price by N - strong earnings growth helped achieve that price far ahead of schedule in august N - the stock has since trading around $ N a share last week and closing yesterday at $ N in national over-the-counter trading - but mr. sim has set a fresh target of $ N a share by the end of - reaching that goal says robert t. applied 's chief financial officer will require efficient reinvestment of cash by applied and of its healthy N N rate of return on operating capital - in barry wright mr. sim sees a situation very similar to the one he faced when he joined applied as president and chief operating officer in N - applied then a closely held company was under the management of its controlling family - while profitable it was n't growing and was n't providing a satisfactory return on invested capital he says - mr. sim is confident that the drive to dominate certain niche markets will work at barry wright as it has at applied - he also an to develop a corporate culture that rewards managers who produce and where is shared - mr. sim considers the new unit 's operations fundamentally sound and adds that barry wright has been fairly successful in moving into markets that have n't interested larger competitors - with a little patience these businesses will perform very mr. sim says - within about six months things will be moving in the right direction he predicts - mr. sim figures it will be easier to turn barry wright around since he 's now in the driver 's seat - when he came to applied i did n't have the power to execute as i do today he says - he was named chief executive officer of applied in N and became chairman last november - at applied mr. sim set growth as his first objective - he took the company public in an offering that applied about $ N million which helped launch the company 's acquisition program - sales climbed to an estimated $ N million in fiscal N ended aug. N from $ N million in fiscal N - the company expects that earnings which have marched steadily upward in recent years reached about $ N million or $ N a share in the fiscal year just ended up from $ N million in fiscal N and $ N million in N diff --git a/test/data_for_tests/people_infer.txt b/test/data_for_tests/people_infer.txt deleted file mode 100644 index 639ea413..00000000 --- a/test/data_for_tests/people_infer.txt +++ /dev/null @@ -1,2 +0,0 @@ -迈向充满希望的新世纪——一九九八年新年讲话 -（附图片1张） \ No newline at end of file diff --git a/test/data_for_tests/sample_snli.jsonl b/test/data_for_tests/sample_snli.jsonl new file mode 100644 index 00000000..e62856ac --- /dev/null +++ b/test/data_for_tests/sample_snli.jsonl @@ -0,0 +1,3 @@ +{"annotator_labels": ["neutral"], "captionID": "3416050480.jpg#4", "gold_label": "neutral", "pairID": "3416050480.jpg#4r1n", "sentence1": "A person on a horse jumps over a broken down airplane.", "sentence1_binary_parse": "( ( ( A person ) ( on ( a horse ) ) ) ( ( jumps ( over ( a ( broken ( down airplane ) ) ) ) ) . ) )", "sentence1_parse": "(ROOT (S (NP (NP (DT A) (NN person)) (PP (IN on) (NP (DT a) (NN horse)))) (VP (VBZ jumps) (PP (IN over) (NP (DT a) (JJ broken) (JJ down) (NN airplane)))) (. .)))", "sentence2": "A person is training his horse for a competition.", "sentence2_binary_parse": "( ( A person ) ( ( is ( ( training ( his horse ) ) ( for ( a competition ) ) ) ) . ) )", "sentence2_parse": "(ROOT (S (NP (DT A) (NN person)) (VP (VBZ is) (VP (VBG training) (NP (PRP$ his) (NN horse)) (PP (IN for) (NP (DT a) (NN competition))))) (. .)))"} +{"annotator_labels": ["contradiction"], "captionID": "3416050480.jpg#4", "gold_label": "contradiction", "pairID": "3416050480.jpg#4r1c", "sentence1": "A person on a horse jumps over a broken down airplane.", "sentence1_binary_parse": "( ( ( A person ) ( on ( a horse ) ) ) ( ( jumps ( over ( a ( broken ( down airplane ) ) ) ) ) . ) )", "sentence1_parse": "(ROOT (S (NP (NP (DT A) (NN person)) (PP (IN on) (NP (DT a) (NN horse)))) (VP (VBZ jumps) (PP (IN over) (NP (DT a) (JJ broken) (JJ down) (NN airplane)))) (. .)))", "sentence2": "A person is at a diner, ordering an omelette.", "sentence2_binary_parse": "( ( A person ) ( ( ( ( is ( at ( a diner ) ) ) , ) ( ordering ( an omelette ) ) ) . ) )", "sentence2_parse": "(ROOT (S (NP (DT A) (NN person)) (VP (VBZ is) (PP (IN at) (NP (DT a) (NN diner))) (, ,) (S (VP (VBG ordering) (NP (DT an) (NN omelette))))) (. .)))"} +{"annotator_labels": ["entailment"], "captionID": "3416050480.jpg#4", "gold_label": "entailment", "pairID": "3416050480.jpg#4r1e", "sentence1": "A person on a horse jumps over a broken down airplane.", "sentence1_binary_parse": "( ( ( A person ) ( on ( a horse ) ) ) ( ( jumps ( over ( a ( broken ( down airplane ) ) ) ) ) . ) )", "sentence1_parse": "(ROOT (S (NP (NP (DT A) (NN person)) (PP (IN on) (NP (DT a) (NN horse)))) (VP (VBZ jumps) (PP (IN over) (NP (DT a) (JJ broken) (JJ down) (NN airplane)))) (. .)))", "sentence2": "A person is outdoors, on a horse.", "sentence2_binary_parse": "( ( A person ) ( ( ( ( is outdoors ) , ) ( on ( a horse ) ) ) . ) )", "sentence2_parse": "(ROOT (S (NP (DT A) (NN person)) (VP (VBZ is) (ADVP (RB outdoors)) (, ,) (PP (IN on) (NP (DT a) (NN horse)))) (. .)))"} \ No newline at end of file diff --git a/test/data_for_tests/word2vec_test.txt b/test/data_for_tests/word2vec_test.txt new file mode 100644 index 00000000..c16170f2 --- /dev/null +++ b/test/data_for_tests/word2vec_test.txt @@ -0,0 +1,7 @@ +5 50 +the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581 +of 0.70853 0.57088 -0.4716 0.18048 0.54449 0.72603 0.18157 -0.52393 0.10381 -0.17566 0.078852 -0.36216 -0.11829 -0.83336 0.11917 -0.16605 0.061555 -0.012719 -0.56623 0.013616 0.22851 -0.14396 -0.067549 -0.38157 -0.23698 -1.7037 -0.86692 -0.26704 -0.2589 0.1767 3.8676 -0.1613 -0.13273 -0.68881 0.18444 0.0052464 -0.33874 -0.078956 0.24185 0.36576 -0.34727 0.28483 0.075693 -0.062178 -0.38988 0.22902 -0.21617 -0.22562 -0.093918 -0.80375 +to 0.68047 -0.039263 0.30186 -0.17792 0.42962 0.032246 -0.41376 0.13228 -0.29847 -0.085253 0.17118 0.22419 -0.10046 -0.43653 0.33418 0.67846 0.057204 -0.34448 -0.42785 -0.43275 0.55963 0.10032 0.18677 -0.26854 0.037334 -2.0932 0.22171 -0.39868 0.20912 -0.55725 3.8826 0.47466 -0.95658 -0.37788 0.20869 -0.32752 0.12751 0.088359 0.16351 -0.21634 -0.094375 0.018324 0.21048 -0.03088 -0.19722 0.082279 -0.09434 -0.073297 -0.064699 -0.26044 +and 0.26818 0.14346 -0.27877 0.016257 0.11384 0.69923 -0.51332 -0.47368 -0.33075 -0.13834 0.2702 0.30938 -0.45012 -0.4127 -0.09932 0.038085 0.029749 0.10076 -0.25058 -0.51818 0.34558 0.44922 0.48791 -0.080866 -0.10121 -1.3777 -0.10866 -0.23201 0.012839 -0.46508 3.8463 0.31362 0.13643 -0.52244 0.3302 0.33707 -0.35601 0.32431 0.12041 0.3512 -0.069043 0.36885 0.25168 -0.24517 0.25381 0.1367 -0.31178 -0.6321 -0.25028 -0.38097 +in 0.33042 0.24995 -0.60874 0.10923 0.036372 0.151 -0.55083 -0.074239 -0.092307 -0.32821 0.09598 -0.82269 -0.36717 -0.67009 0.42909 0.016496 -0.23573 0.12864 -1.0953 0.43334 0.57067 -0.1036 0.20422 0.078308 -0.42795 -1.7984 -0.27865 0.11954 -0.12689 0.031744 3.8631 -0.17786 -0.082434 -0.62698 0.26497 -0.057185 -0.073521 0.46103 0.30862 0.12498 -0.48609 -0.0080272 0.031184 -0.36576 -0.42699 0.42164 -0.11666 -0.50703 -0.027273 -0.53285 +a 0.21705 0.46515 -0.46757 0.10082 1.0135 0.74845 -0.53104 -0.26256 0.16812 0.13182 -0.24909 -0.44185 -0.21739 0.51004 0.13448 -0.43141 -0.03123 0.20674 -0.78138 -0.20148 -0.097401 0.16088 -0.61836 -0.18504 -0.12461 -2.2526 -0.22321 0.5043 0.32257 0.15313 3.9636 -0.71365 -0.67012 0.28388 0.21738 0.14433 0.25926 0.23434 0.4274 -0.44451 0.13813 0.36973 -0.64289 0.024142 -0.039315 -0.26037 0.12017 -0.043782 0.41013 0.1796 \ No newline at end of file diff --git a/test/data_for_tests/zh_sample.conllx b/test/data_for_tests/zh_sample.conllx new file mode 100644 index 00000000..dee802ef --- /dev/null +++ b/test/data_for_tests/zh_sample.conllx @@ -0,0 +1,100 @@ +1 上海 _ NR NR _ 3 nsubj _ _ +2 积极 _ AD AD _ 3 advmod _ _ +3 准备 _ VV VV _ 0 root _ _ +4 迎接 _ VV VV _ 3 ccomp _ _ +5 欧元 _ NN NN _ 6 nn _ _ +6 诞生 _ NN NN _ 4 dobj _ _ + +1 新华社 _ NR NR _ 7 dep _ _ +2 上海 _ NR NR _ 7 dep _ _ +3 十二月 _ NT NT _ 7 dep _ _ +4 三十日 _ NT NT _ 7 dep _ _ +5 电 _ NN NN _ 7 dep _ _ +6 （ _ PU PU _ 7 punct _ _ +7 记者 _ NN NN _ 0 root _ _ +8 潘清 _ NR NR _ 7 dep _ _ +9 ） _ PU PU _ 7 punct _ _ + +1 即将 _ AD AD _ 2 advmod _ _ +2 诞生 _ VV VV _ 4 rcmod _ _ +3 的 _ DEC DEC _ 2 cpm _ _ +4 欧元 _ NN NN _ 6 nsubj _ _ +5 ， _ PU PU _ 6 punct _ _ +6 引起 _ VV VV _ 0 root _ _ +7 了 _ AS AS _ 6 asp _ _ +8 上海 _ NR NR _ 14 nn _ _ +9 这 _ DT DT _ 14 det _ _ +10 个 _ M M _ 9 clf _ _ +11 中国 _ NR NR _ 13 nn _ _ +12 金融 _ NN NN _ 13 nn _ _ +13 中心 _ NN NN _ 14 nn _ _ +14 城市 _ NN NN _ 16 assmod _ _ +15 的 _ DEG DEG _ 14 assm _ _ +16 关注 _ NN NN _ 6 dobj _ _ +17 。 _ PU PU _ 6 punct _ _ + +1 上海 _ NR NR _ 2 nn _ _ +2 银行界 _ NN NN _ 4 nsubj _ _ +3 纷纷 _ AD AD _ 4 advmod _ _ +4 推出 _ VV VV _ 0 root _ _ +5 了 _ AS AS _ 4 asp _ _ +6 与 _ P P _ 8 prep _ _ +7 之 _ PN PN _ 6 pobj _ _ +8 相关 _ VA VA _ 15 rcmod _ _ +9 的 _ DEC DEC _ 8 cpm _ _ +10 外汇 _ NN NN _ 15 nn _ _ +11 业务 _ NN NN _ 15 nn _ _ +12 品种 _ NN NN _ 15 conj _ _ +13 和 _ CC CC _ 15 cc _ _ +14 服务 _ NN NN _ 15 nn _ _ +15 举措 _ NN NN _ 4 dobj _ _ +16 ， _ PU PU _ 4 punct _ _ +17 积极 _ AD AD _ 18 advmod _ _ +18 准备 _ VV VV _ 4 dep _ _ +19 启动 _ VV VV _ 18 ccomp _ _ +20 欧元 _ NN NN _ 21 nn _ _ +21 业务 _ NN NN _ 19 dobj _ _ +22 。 _ PU PU _ 4 punct _ _ + +1 一些 _ CD CD _ 8 nummod _ _ +2 热衷于 _ VV VV _ 8 rcmod _ _ +3 个人 _ NN NN _ 5 nn _ _ +4 外汇 _ NN NN _ 5 nn _ _ +5 交易 _ NN NN _ 2 dobj _ _ +6 的 _ DEC DEC _ 2 cpm _ _ +7 上海 _ NR NR _ 8 nn _ _ +8 市民 _ NN NN _ 13 nsubj _ _ +9 ， _ PU PU _ 13 punct _ _ +10 也 _ AD AD _ 13 advmod _ _ +11 对 _ P P _ 13 prep _ _ +12 欧元 _ NN NN _ 11 pobj _ _ +13 表示 _ VV VV _ 0 root _ _ +14 出 _ VV VV _ 13 rcomp _ _ +15 极 _ AD AD _ 16 advmod _ _ +16 大 _ VA VA _ 18 rcmod _ _ +17 的 _ DEC DEC _ 16 cpm _ _ +18 兴趣 _ NN NN _ 13 dobj _ _ +19 。 _ PU PU _ 13 punct _ _ + +1 继 _ P P _ 38 prep _ _ +2 上海 _ NR NR _ 6 nn _ _ +3 大众 _ NR NR _ 6 nn _ _ +4 汽车 _ NN NN _ 6 nn _ _ +5 有限 _ JJ JJ _ 6 amod _ _ +6 公司 _ NN NN _ 13 nsubj _ _ +7 十八日 _ NT NT _ 13 tmod _ _ +8 在 _ P P _ 13 prep _ _ +9 中国 _ NR NR _ 10 nn _ _ +10 银行 _ NN NN _ 12 nn _ _ +11 上海 _ NR NR _ 12 nn _ _ +12 分行 _ NN NN _ 8 pobj _ _ +13 开立 _ VV VV _ 19 lccomp _ _ +14 上海 _ NR NR _ 16 dep _ _ +15 第一 _ OD OD _ 16 ordmod _ _ +16 个 _ M M _ 18 clf _ _ +17 欧元 _ NN NN _ 18 nn _ _ +18 帐户 _ NN NN _ 13 dobj _ _ +19 后 _ LC LC _ 1 plmod _ _ +20 ， _ PU PU _ 38 punct _ _ +21 工商 _ NN NN _ 28 nn _ _ +22 银行 _ NN NN _ 28 conj _ _ diff --git a/test/io/config b/test/io/config deleted file mode 100644 index 5ff9eacf..00000000 --- a/test/io/config +++ /dev/null @@ -1,62 +0,0 @@ -[test] -x = 1 - -y = 2 - -z = 3 -#this is an example -input = [1,2,3] - -text = "this is text" - -doubles = 0.8 - -tt = 0.5 - -test = 105 - -str = "this is a str" - -double = 0.5 - - -[t] -x = "this is an test section" - - - -[test-case-2] -double = 0.5 - -doubles = 0.8 - -tt = 0.5 - -test = 105 - -str = "this is a str" - -[another-test] -doubles = 0.8 - -tt = 0.5 - -test = 105 - -str = "this is a str" - -double = 0.5 - - -[one-another-test] -doubles = 0.8 - -tt = 0.5 - -test = 105 - -str = "this is a str" - -double = 0.5 - - diff --git a/test/io/test_config_saver.py b/test/io/test_config_saver.py deleted file mode 100644 index f29097c5..00000000 --- a/test/io/test_config_saver.py +++ /dev/null @@ -1,112 +0,0 @@ -import os -import unittest - -from fastNLP.io.config_io import ConfigSection, ConfigLoader, ConfigSaver - - -class TestConfigSaver(unittest.TestCase): - def test_case_1(self): - config_file_dir = "test/io/" - config_file_name = "config" - config_file_path = os.path.join(config_file_dir, config_file_name) - - tmp_config_file_path = os.path.join(config_file_dir, "tmp_config") - - with open(config_file_path, "r") as f: - lines = f.readlines() - - standard_section = ConfigSection() - t_section = ConfigSection() - ConfigLoader().load_config(config_file_path, {"test": standard_section, "t": t_section}) - - config_saver = ConfigSaver(config_file_path) - - section = ConfigSection() - section["doubles"] = 0.8 - section["tt"] = 0.5 - section["test"] = 105 - section["str"] = "this is a str" - - test_case_2_section = section - test_case_2_section["double"] = 0.5 - - for k in section.__dict__.keys(): - standard_section[k] = section[k] - - config_saver.save_config_file("test", section) - config_saver.save_config_file("another-test", section) - config_saver.save_config_file("one-another-test", section) - config_saver.save_config_file("test-case-2", section) - - test_section = ConfigSection() - at_section = ConfigSection() - another_test_section = ConfigSection() - one_another_test_section = ConfigSection() - a_test_case_2_section = ConfigSection() - - ConfigLoader().load_config(config_file_path, {"test": test_section, - "another-test": another_test_section, - "t": at_section, - "one-another-test": one_another_test_section, - "test-case-2": a_test_case_2_section}) - - assert test_section == standard_section - assert at_section == t_section - assert another_test_section == section - assert one_another_test_section == section - assert a_test_case_2_section == test_case_2_section - - config_saver.save_config_file("test", section) - - with open(config_file_path, "w") as f: - f.writelines(lines) - - with open(tmp_config_file_path, "w") as f: - f.write('[test]\n') - f.write('this is an fault example\n') - - tmp_config_saver = ConfigSaver(tmp_config_file_path) - try: - tmp_config_saver._read_section() - except Exception as e: - pass - os.remove(tmp_config_file_path) - - try: - tmp_config_saver = ConfigSaver("file-NOT-exist") - except Exception as e: - pass - - def test_case_2(self): - config = "[section_A]\n[section_B]\n" - - with open("./test.cfg", "w", encoding="utf-8") as f: - f.write(config) - saver = ConfigSaver("./test.cfg") - - section = ConfigSection() - section["doubles"] = 0.8 - section["tt"] = [1, 2, 3] - section["test"] = 105 - section["str"] = "this is a str" - - saver.save_config_file("section_A", section) - - os.system("rm ./test.cfg") - - def test_case_3(self): - config = "[section_A]\ndoubles = 0.9\ntt = [1, 2, 3]\n[section_B]\n" - - with open("./test.cfg", "w", encoding="utf-8") as f: - f.write(config) - saver = ConfigSaver("./test.cfg") - - section = ConfigSection() - section["doubles"] = 0.8 - section["tt"] = [1, 2, 3] - section["test"] = 105 - section["str"] = "this is a str" - - saver.save_config_file("section_A", section) - - os.system("rm ./test.cfg") diff --git a/test/io/test_dataset_loader.py b/test/io/test_dataset_loader.py index cf38c973..12d352b1 100644 --- a/test/io/test_dataset_loader.py +++ b/test/io/test_dataset_loader.py @@ -1,24 +1,30 @@ import unittest -from fastNLP.io.dataset_loader import Conll2003Loader +from fastNLP.io import Conll2003Loader, PeopleDailyCorpusLoader, CSVLoader, SNLILoader, JsonLoader class TestDatasetLoader(unittest.TestCase): - - def test_case_1(self): - ''' + + def test_Conll2003Loader(self): + """ Test the the loader of Conll2003 dataset - ''' - + """ dataset_path = "test/data_for_tests/conll_2003_example.txt" loader = Conll2003Loader() dataset_2003 = loader.load(dataset_path) - - for item in dataset_2003: - len0 = len(item["label0_list"]) - len1 = len(item["label1_list"]) - len2 = len(item["label2_list"]) - lentoken = len(item["token_list"]) - self.assertNotEqual(len0, 0) - self.assertEqual(len0, len1) - self.assertEqual(len1, len2) + + def test_PeopleDailyCorpusLoader(self): + data_set = PeopleDailyCorpusLoader().load("test/data_for_tests/people_daily_raw.txt") + + def test_CSVLoader(self): + ds = CSVLoader(sep='\t', headers=['words', 'label']) \ + .load('test/data_for_tests/tutorial_sample_dataset.csv') + assert len(ds) > 0 + + def test_SNLILoader(self): + ds = SNLILoader().load('test/data_for_tests/sample_snli.jsonl') + assert len(ds) == 3 + + def test_JsonLoader(self): + ds = JsonLoader().load('test/data_for_tests/sample_snli.jsonl') + assert len(ds) == 3 diff --git a/test/io/test_embed_loader.py b/test/io/test_embed_loader.py index 60e3710e..ff8ecfcf 100644 --- a/test/io/test_embed_loader.py +++ b/test/io/test_embed_loader.py @@ -1,12 +1,50 @@ import unittest +import numpy as np -from fastNLP.core.vocabulary import Vocabulary -from fastNLP.io.embed_loader import EmbedLoader +from fastNLP import Vocabulary +from fastNLP.io import EmbedLoader class TestEmbedLoader(unittest.TestCase): - def test_case(self): + def test_load_with_vocab(self): vocab = Vocabulary() - vocab.update(["the", "in", "I", "to", "of", "hahaha"]) - embedding = EmbedLoader().fast_load_embedding(50, "test/data_for_tests/glove.6B.50d_test.txt", vocab) - self.assertEqual(tuple(embedding.shape), (len(vocab), 50)) + glove = "test/data_for_tests/glove.6B.50d_test.txt" + word2vec = "test/data_for_tests/word2vec_test.txt" + vocab.add_word('the') + vocab.add_word('none') + g_m = EmbedLoader.load_with_vocab(glove, vocab) + self.assertEqual(g_m.shape, (4, 50)) + w_m = EmbedLoader.load_with_vocab(word2vec, vocab, normalize=True) + self.assertEqual(w_m.shape, (4, 50)) + self.assertAlmostEqual(np.linalg.norm(w_m, axis=1).sum(), 4) + + def test_load_without_vocab(self): + words = ['the', 'of', 'in', 'a', 'to', 'and'] + glove = "test/data_for_tests/glove.6B.50d_test.txt" + word2vec = "test/data_for_tests/word2vec_test.txt" + g_m, vocab = EmbedLoader.load_without_vocab(glove) + self.assertEqual(g_m.shape, (8, 50)) + for word in words: + self.assertIn(word, vocab) + w_m, vocab = EmbedLoader.load_without_vocab(word2vec, normalize=True) + self.assertEqual(w_m.shape, (8, 50)) + self.assertAlmostEqual(np.linalg.norm(w_m, axis=1).sum(), 8) + for word in words: + self.assertIn(word, vocab) + # no unk + w_m, vocab = EmbedLoader.load_without_vocab(word2vec, normalize=True, unknown=None) + self.assertEqual(w_m.shape, (7, 50)) + self.assertAlmostEqual(np.linalg.norm(w_m, axis=1).sum(), 7) + for word in words: + self.assertIn(word, vocab) + + def test_read_all_glove(self): + pass + # TODO + # 这是可以运行的，但是总数少于行数，应该是由于glove有重复的word + # path = '/where/to/read/full/glove' + # init_embed, vocab = EmbedLoader.load_without_vocab(path, error='strict') + # print(init_embed.shape) + # print(init_embed.mean()) + # print(np.isnan(init_embed).sum()) + # print(len(vocab)) diff --git a/test/models/__init__.py b/test/models/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/models/model_runner.py b/test/models/model_runner.py new file mode 100644 index 00000000..405aa7d6 --- /dev/null +++ b/test/models/model_runner.py @@ -0,0 +1,156 @@ +""" +此模块可以非常方便的测试模型。 +若你的模型属于：文本分类，序列标注，自然语言推理（NLI），可以直接使用此模块测试 +若模型不属于上述类别，也可以自己准备假数据，设定loss和metric进行测试 + +此模块的测试仅保证模型能使用fastNLP进行训练和测试，不测试模型实际性能 + +Example:: + + # import 全大写变量... + from model_runner import * + + # 测试一个文本分类模型 + init_emb = (VOCAB_SIZE, 50) + model = SomeModel(init_emb, num_cls=NUM_CLS) + RUNNER.run_model_with_task(TEXT_CLS, model) + + # 序列标注模型 + RUNNER.run_model_with_task(POS_TAGGING, model) + + # NLI模型 + RUNNER.run_model_with_task(NLI, model) + + # 自定义模型 + RUNNER.run_model(model, data=get_mydata(), + loss=Myloss(), metrics=Mymetric()) +""" +from fastNLP import Trainer, Tester, DataSet, Callback +from fastNLP import AccuracyMetric +from fastNLP import CrossEntropyLoss +from fastNLP.core.const import Const as C +from random import randrange + +VOCAB_SIZE = 100 +NUM_CLS = 100 +MAX_LEN = 10 +N_SAMPLES = 100 +N_EPOCHS = 1 +BATCH_SIZE = 5 + +TEXT_CLS = 'text_cls' +POS_TAGGING = 'pos_tagging' +NLI = 'nli' + +class ModelRunner(): + class Checker(Callback): + def on_backward_begin(self, loss): + assert loss.to('cpu').numpy().isfinate() + + def gen_seq(self, length, vocab_size): + """generate fake sequence indexes with given length""" + # reserve 0 for padding + return [randrange(1, vocab_size) for _ in range(length)] + + def gen_var_seq(self, max_len, vocab_size): + """generate fake sequence indexes in variant length""" + length = randrange(3, max_len) # at least 3 words in a seq + return self.gen_seq(length, vocab_size) + + def prepare_text_classification_data(self): + index = 'index' + ds = DataSet({index: list(range(N_SAMPLES))}) + ds.apply_field(lambda x: self.gen_var_seq(MAX_LEN, VOCAB_SIZE), + field_name=index, new_field_name=C.INPUT, + is_input=True) + ds.apply_field(lambda x: randrange(NUM_CLS), + field_name=index, new_field_name=C.TARGET, + is_target=True) + ds.apply_field(len, C.INPUT, C.INPUT_LEN, + is_input=True) + return ds + + def prepare_pos_tagging_data(self): + index = 'index' + ds = DataSet({index: list(range(N_SAMPLES))}) + ds.apply_field(lambda x: self.gen_var_seq(MAX_LEN, VOCAB_SIZE), + field_name=index, new_field_name=C.INPUT, + is_input=True) + ds.apply_field(lambda x: self.gen_seq(len(x), NUM_CLS), + field_name=C.INPUT, new_field_name=C.TARGET, + is_target=True) + ds.apply_field(len, C.INPUT, C.INPUT_LEN, + is_input=True, is_target=True) + return ds + + def prepare_nli_data(self): + index = 'index' + ds = DataSet({index: list(range(N_SAMPLES))}) + ds.apply_field(lambda x: self.gen_var_seq(MAX_LEN, VOCAB_SIZE), + field_name=index, new_field_name=C.INPUTS(0), + is_input=True) + ds.apply_field(lambda x: self.gen_var_seq(MAX_LEN, VOCAB_SIZE), + field_name=index, new_field_name=C.INPUTS(1), + is_input=True) + ds.apply_field(lambda x: randrange(NUM_CLS), + field_name=index, new_field_name=C.TARGET, + is_target=True) + ds.apply_field(len, C.INPUTS(0), C.INPUT_LENS(0), + is_input=True, is_target=True) + ds.apply_field(len, C.INPUTS(1), C.INPUT_LENS(1), + is_input = True, is_target = True) + ds.set_input(C.INPUTS(0), C.INPUTS(1)) + ds.set_target(C.TARGET) + return ds + + def run_text_classification(self, model, data=None): + if data is None: + data = self.prepare_text_classification_data() + loss = CrossEntropyLoss(pred=C.OUTPUT, target=C.TARGET) + metric = AccuracyMetric(pred=C.OUTPUT, target=C.TARGET) + self.run_model(model, data, loss, metric) + + def run_pos_tagging(self, model, data=None): + if data is None: + data = self.prepare_pos_tagging_data() + loss = CrossEntropyLoss(pred=C.OUTPUT, target=C.TARGET, padding_idx=0) + metric = AccuracyMetric(pred=C.OUTPUT, target=C.TARGET, seq_len=C.INPUT_LEN) + self.run_model(model, data, loss, metric) + + def run_nli(self, model, data=None): + if data is None: + data = self.prepare_nli_data() + loss = CrossEntropyLoss(pred=C.OUTPUT, target=C.TARGET) + metric = AccuracyMetric(pred=C.OUTPUT, target=C.TARGET) + self.run_model(model, data, loss, metric) + + def run_model(self, model, data, loss, metrics): + """run a model, test if it can run with fastNLP""" + print('testing model:', model.__class__.__name__) + tester = Tester(data=data, model=model, metrics=metrics, + batch_size=BATCH_SIZE, verbose=0) + before_train = tester.test() + trainer = Trainer(model=model, train_data=data, dev_data=None, + n_epochs=N_EPOCHS, batch_size=BATCH_SIZE, + loss=loss, + save_path=None, + use_tqdm=False) + trainer.train(load_best_model=False) + after_train = tester.test() + for metric_name, v1 in before_train.items(): + assert metric_name in after_train + # # at least we can sure model params changed, even if we don't know performance + # v2 = after_train[metric_name] + # assert v1 != v2 + + def run_model_with_task(self, task, model): + """run a model with certain task""" + TASKS = { + TEXT_CLS: self.run_text_classification, + POS_TAGGING: self.run_pos_tagging, + NLI: self.run_nli, + } + assert task in TASKS + TASKS[task](model) + +RUNNER = ModelRunner() diff --git a/test/models/test_bert.py b/test/models/test_bert.py new file mode 100644 index 00000000..b2899a89 --- /dev/null +++ b/test/models/test_bert.py @@ -0,0 +1,21 @@ +import unittest + +import torch + +from fastNLP.models.bert import BertModel + + +class TestBert(unittest.TestCase): + def test_bert_1(self): + # model = BertModel.from_pretrained("/home/zyfeng/data/bert-base-chinese") + model = BertModel(vocab_size=32000, hidden_size=768, + num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) + + input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) + input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) + token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) + + all_encoder_layers, pooled_output = model(input_ids, token_type_ids, input_mask) + for layer in all_encoder_layers: + self.assertEqual(tuple(layer.shape), (2, 3, 768)) + self.assertEqual(tuple(pooled_output.shape), (2, 768)) diff --git a/test/models/test_biaffine_parser.py b/test/models/test_biaffine_parser.py index 54935f76..e6fca6a8 100644 --- a/test/models/test_biaffine_parser.py +++ b/test/models/test_biaffine_parser.py @@ -1,87 +1,48 @@ -from fastNLP.models.biaffine_parser import BiaffineParser, ParserLoss, ParserMetric -import fastNLP - import unittest -data_file = """ -1 The _ DET DT _ 3 det _ _ -2 new _ ADJ JJ _ 3 amod _ _ -3 rate _ NOUN NN _ 6 nsubj _ _ -4 will _ AUX MD _ 6 aux _ _ -5 be _ VERB VB _ 6 cop _ _ -6 payable _ ADJ JJ _ 0 root _ _ -7 mask _ ADJ JJ _ 6 punct _ _ -8 mask _ ADJ JJ _ 6 punct _ _ -9 cents _ NOUN NNS _ 4 nmod _ _ -10 from _ ADP IN _ 12 case _ _ -11 seven _ NUM CD _ 12 nummod _ _ -12 cents _ NOUN NNS _ 4 nmod _ _ -13 a _ DET DT _ 14 det _ _ -14 share _ NOUN NN _ 12 nmod:npmod _ _ -15 . _ PUNCT . _ 4 punct _ _ - -1 The _ DET DT _ 3 det _ _ -2 new _ ADJ JJ _ 3 amod _ _ -3 rate _ NOUN NN _ 6 nsubj _ _ -4 will _ AUX MD _ 6 aux _ _ -5 be _ VERB VB _ 6 cop _ _ -6 payable _ ADJ JJ _ 0 root _ _ -7 Feb. _ PROPN NNP _ 6 nmod:tmod _ _ -8 15 _ NUM CD _ 7 nummod _ _ -9 . _ PUNCT . _ 6 punct _ _ - -1 A _ DET DT _ 3 det _ _ -2 record _ NOUN NN _ 3 compound _ _ -3 date _ NOUN NN _ 7 nsubjpass _ _ -4 has _ AUX VBZ _ 7 aux _ _ -5 n't _ PART RB _ 7 neg _ _ -6 been _ AUX VBN _ 7 auxpass _ _ -7 set _ VERB VBN _ 0 root _ _ -8 . _ PUNCT . _ 7 punct _ _ - -""" - -def init_data(): - ds = fastNLP.DataSet() - v = {'word_seq': fastNLP.Vocabulary(), - 'pos_seq': fastNLP.Vocabulary(), - 'label_true': fastNLP.Vocabulary()} - data = [] - for line in data_file.split('\n'): - line = line.split() - if len(line) == 0 and len(data) > 0: - data = list(zip(*data)) - ds.append(fastNLP.Instance(word_seq=data[1], - pos_seq=data[4], - arc_true=data[6], - label_true=data[7])) - data = [] - elif len(line) > 0: - data.append(line) - - for name in ['word_seq', 'pos_seq', 'label_true']: - ds.apply(lambda x: ['']+list(x[name]), new_field_name=name) - ds.apply(lambda x: v[name].add_word_lst(x[name])) - - for name in ['word_seq', 'pos_seq', 'label_true']: - ds.apply(lambda x: [v[name].to_index(w) for w in x[name]], new_field_name=name) +import fastNLP +from fastNLP.models.biaffine_parser import BiaffineParser, ParserLoss, ParserMetric +from .model_runner import * + + +def prepare_parser_data(): + index = 'index' + ds = DataSet({index: list(range(N_SAMPLES))}) + ds.apply_field(lambda x: RUNNER.gen_var_seq(MAX_LEN, VOCAB_SIZE), + field_name=index, new_field_name=C.INPUTS(0), + is_input=True) + ds.apply_field(lambda x: RUNNER.gen_seq(len(x), NUM_CLS), + field_name=C.INPUTS(0), new_field_name=C.INPUTS(1), + is_input=True) + # target1 is heads, should in range(0, len(words)) + ds.apply_field(lambda x: RUNNER.gen_seq(len(x), len(x)), + field_name=C.INPUTS(0), new_field_name=C.TARGETS(0), + is_target=True) + ds.apply_field(lambda x: RUNNER.gen_seq(len(x), NUM_CLS), + field_name=C.INPUTS(0), new_field_name=C.TARGETS(1), + is_target=True) + ds.apply_field(len, field_name=C.INPUTS(0), new_field_name=C.INPUT_LEN, + is_input=True, is_target=True) + return ds - ds.apply(lambda x: [0]+list(map(int, x['arc_true'])), new_field_name='arc_true') - ds.apply(lambda x: len(x['word_seq']), new_field_name='seq_lens') - ds.set_input('word_seq', 'pos_seq', 'seq_lens', flag=True) - ds.set_target('arc_true', 'label_true', 'seq_lens', flag=True) - return ds, v['word_seq'], v['pos_seq'], v['label_true'] class TestBiaffineParser(unittest.TestCase): def test_train(self): - ds, v1, v2, v3 = init_data() - model = BiaffineParser(word_vocab_size=len(v1), word_emb_dim=30, - pos_vocab_size=len(v2), pos_emb_dim=30, - num_label=len(v3), use_var_lstm=True) - trainer = fastNLP.Trainer(model=model, train_data=ds, dev_data=ds, - loss=ParserLoss(), metrics=ParserMetric(), metric_key='UAS', - n_epochs=10, use_cuda=False, use_tqdm=False) - trainer.train(load_best_model=False) - -if __name__ == '__main__': - unittest.main() \ No newline at end of file + model = BiaffineParser(init_embed=(VOCAB_SIZE, 10), + pos_vocab_size=VOCAB_SIZE, pos_emb_dim=10, + rnn_hidden_size=10, + arc_mlp_size=10, + label_mlp_size=10, + num_label=NUM_CLS, encoder='var-lstm') + ds = prepare_parser_data() + RUNNER.run_model(model, ds, loss=ParserLoss(), metrics=ParserMetric()) + + def test_train2(self): + model = BiaffineParser(init_embed=(VOCAB_SIZE, 10), + pos_vocab_size=VOCAB_SIZE, pos_emb_dim=10, + rnn_hidden_size=16, + arc_mlp_size=10, + label_mlp_size=10, + num_label=NUM_CLS, encoder='transformer') + ds = prepare_parser_data() + RUNNER.run_model(model, ds, loss=ParserLoss(), metrics=ParserMetric()) diff --git a/test/models/test_cnn_text_classification.py b/test/models/test_cnn_text_classification.py new file mode 100644 index 00000000..b83b7bad --- /dev/null +++ b/test/models/test_cnn_text_classification.py @@ -0,0 +1,18 @@ + +import unittest + +from .model_runner import * +from fastNLP.models.cnn_text_classification import CNNText + + +class TestCNNText(unittest.TestCase): + def test_case1(self): + # 测试能否正常运行CNN + init_emb = (VOCAB_SIZE, 30) + model = CNNText(init_emb, + NUM_CLS, + kernel_nums=(1, 3, 5), + kernel_sizes=(2, 2, 2), + padding=0, + dropout=0.5) + RUNNER.run_model_with_task(TEXT_CLS, model) diff --git a/test/models/test_sequence_labeling.py b/test/models/test_sequence_labeling.py new file mode 100644 index 00000000..3a70e381 --- /dev/null +++ b/test/models/test_sequence_labeling.py @@ -0,0 +1,36 @@ + + +import unittest + +from .model_runner import * +from fastNLP.models.sequence_labeling import SeqLabeling, AdvSeqLabel +from fastNLP.core.losses import LossInForward + +class TesSeqLabel(unittest.TestCase): + def test_case1(self): + # 测试能否正常运行CNN + init_emb = (VOCAB_SIZE, 30) + model = SeqLabeling(init_emb, + hidden_size=30, + num_classes=NUM_CLS) + + data = RUNNER.prepare_pos_tagging_data() + data.set_input('target') + loss = LossInForward() + metric = AccuracyMetric(pred=C.OUTPUT, target=C.TARGET, seq_len=C.INPUT_LEN) + RUNNER.run_model(model, data, loss, metric) + + +class TesAdvSeqLabel(unittest.TestCase): + def test_case1(self): + # 测试能否正常运行CNN + init_emb = (VOCAB_SIZE, 30) + model = AdvSeqLabel(init_emb, + hidden_size=30, + num_classes=NUM_CLS) + + data = RUNNER.prepare_pos_tagging_data() + data.set_input('target') + loss = LossInForward() + metric = AccuracyMetric(pred=C.OUTPUT, target=C.TARGET, seq_len=C.INPUT_LEN) + RUNNER.run_model(model, data, loss, metric) \ No newline at end of file diff --git a/test/models/test_star_trans.py b/test/models/test_star_trans.py new file mode 100644 index 00000000..eae19b24 --- /dev/null +++ b/test/models/test_star_trans.py @@ -0,0 +1,16 @@ +from .model_runner import * +from fastNLP.models.star_transformer import STNLICls, STSeqCls, STSeqLabel + + +# add star-transformer tests, for 3 kinds of tasks. +def test_cls(): + model = STSeqCls((VOCAB_SIZE, 10), NUM_CLS, dropout=0) + RUNNER.run_model_with_task(TEXT_CLS, model) + +def test_nli(): + model = STNLICls((VOCAB_SIZE, 10), NUM_CLS, dropout=0) + RUNNER.run_model_with_task(NLI, model) + +def test_seq_label(): + model = STSeqLabel((VOCAB_SIZE, 10), NUM_CLS, dropout=0) + RUNNER.run_model_with_task(POS_TAGGING, model) diff --git a/test/modules/decoder/test_CRF.py b/test/modules/decoder/test_CRF.py index 0fc331dc..5dec7d47 100644 --- a/test/modules/decoder/test_CRF.py +++ b/test/modules/decoder/test_CRF.py @@ -5,7 +5,7 @@ import unittest class TestCRF(unittest.TestCase): def test_case1(self): # 检查allowed_transitions()能否正确使用 - from fastNLP.modules.decoder.CRF import allowed_transitions + from fastNLP.modules.decoder.crf import allowed_transitions id2label = {0: 'B', 1: 'I', 2:'O'} expected_res = {(0, 0), (0, 1), (0, 2), (0, 4), (1, 0), (1, 1), (1, 2), (1, 4), (2, 0), (2, 2), @@ -43,7 +43,7 @@ class TestCRF(unittest.TestCase): # 测试CRF能否避免解码出非法跃迁, 使用allennlp做了验证。 pass # import torch - # from fastNLP.modules.decoder.CRF import seq_len_to_byte_mask + # from fastNLP.modules.decoder.crf import seq_len_to_byte_mask # # labels = ['O'] # for label in ['X', 'Y']: @@ -63,10 +63,10 @@ class TestCRF(unittest.TestCase): # mask = seq_len_to_byte_mask(seq_lens) # allen_res = allen_CRF.viterbi_tags(logits, mask) # - # from fastNLP.modules.decoder.CRF import ConditionalRandomField, allowed_transitions + # from fastNLP.modules.decoder.crf import ConditionalRandomField, allowed_transitions # fast_CRF = ConditionalRandomField(num_tags=num_tags, allowed_transitions=allowed_transitions(id2label)) # fast_CRF.trans_m = trans_m - # fast_res = fast_CRF.viterbi_decode(logits, mask, get_score=True) + # fast_res = fast_CRF.viterbi_decode(logits, mask, get_score=True, unpad=True) # # score equal # self.assertListEqual([score for _, score in allen_res], fast_res[1]) # # seq equal @@ -91,14 +91,38 @@ class TestCRF(unittest.TestCase): # mask = seq_len_to_byte_mask(seq_lens) # allen_res = allen_CRF.viterbi_tags(logits, mask) # - # from fastNLP.modules.decoder.CRF import ConditionalRandomField, allowed_transitions + # from fastNLP.modules.decoder.crf import ConditionalRandomField, allowed_transitions # fast_CRF = ConditionalRandomField(num_tags=num_tags, allowed_transitions=allowed_transitions(id2label, # encoding_type='BMES')) # fast_CRF.trans_m = trans_m - # fast_res = fast_CRF.viterbi_decode(logits, mask, get_score=True) + # fast_res = fast_CRF.viterbi_decode(logits, mask, get_score=True, unpad=True) # # score equal # self.assertListEqual([score for _, score in allen_res], fast_res[1]) # # seq equal # self.assertListEqual([_ for _, score in allen_res], fast_res[0]) + def test_case3(self): + # 测试crf的loss不会出现负数 + import torch + from fastNLP.modules.decoder.crf import ConditionalRandomField + from fastNLP.core.utils import seq_len_to_mask + from torch import optim + from torch import nn + num_tags, include_start_end_trans = 4, True + num_samples = 4 + lengths = torch.randint(3, 50, size=(num_samples, )).long() + max_len = lengths.max() + tags = torch.randint(num_tags, size=(num_samples, max_len)) + masks = seq_len_to_mask(lengths) + feats = nn.Parameter(torch.randn(num_samples, max_len, num_tags)) + crf = ConditionalRandomField(num_tags, include_start_end_trans) + optimizer = optim.SGD([param for param in crf.parameters() if param.requires_grad] + [feats], lr=0.1) + for _ in range(10): + loss = crf(feats, tags, masks).mean() + optimizer.zero_grad() + loss.backward() + optimizer.step() + if _%1000==0: + print(loss) + self.assertGreater(loss.item(), 0, "CRF loss cannot be less than 0.") diff --git a/test/modules/test_char_embedding.py b/test/modules/test_char_encoder.py similarity index 81% rename from test/modules/test_char_embedding.py rename to test/modules/test_char_encoder.py index 07def64a..cf3ec15e 100644 --- a/test/modules/test_char_embedding.py +++ b/test/modules/test_char_encoder.py @@ -2,7 +2,7 @@ import unittest import torch -from fastNLP.modules.encoder.char_embedding import ConvCharEmbedding, LSTMCharEmbedding +from fastNLP.modules.encoder.char_encoder import ConvolutionCharEncoder, LSTMCharEncoder class TestCharEmbed(unittest.TestCase): @@ -13,14 +13,14 @@ class TestCharEmbed(unittest.TestCase): x = torch.Tensor(batch_size, char_emb, word_length) x = x.transpose(1, 2) - cce = ConvCharEmbedding(char_emb) + cce = ConvolutionCharEncoder(char_emb) y = cce(x) self.assertEqual(tuple(x.shape), (batch_size, word_length, char_emb)) print("CNN Char Emb input: ", x.shape) self.assertEqual(tuple(y.shape), (batch_size, char_emb, 1)) print("CNN Char Emb output: ", y.shape) # [128, 100] - lce = LSTMCharEmbedding(char_emb) + lce = LSTMCharEncoder(char_emb) o = lce(x) self.assertEqual(tuple(x.shape), (batch_size, word_length, char_emb)) print("LSTM Char Emb input: ", x.shape) diff --git a/test/modules/test_masked_rnn.py b/test/modules/test_masked_rnn.py deleted file mode 100644 index 80f49f33..00000000 --- a/test/modules/test_masked_rnn.py +++ /dev/null @@ -1,27 +0,0 @@ - -import torch -import unittest - -from fastNLP.modules.encoder.masked_rnn import MaskedRNN - -class TestMaskedRnn(unittest.TestCase): - def test_case_1(self): - masked_rnn = MaskedRNN(input_size=1, hidden_size=1, bidirectional=True, batch_first=True) - x = torch.tensor([[[1.0], [2.0]]]) - print(x.size()) - y = masked_rnn(x) - mask = torch.tensor([[[1], [1]]]) - y = masked_rnn(x, mask=mask) - mask = torch.tensor([[[1], [0]]]) - y = masked_rnn(x, mask=mask) - - def test_case_2(self): - masked_rnn = MaskedRNN(input_size=1, hidden_size=1, bidirectional=False, batch_first=True) - x = torch.tensor([[[1.0], [2.0]]]) - print(x.size()) - y = masked_rnn(x) - mask = torch.tensor([[[1], [1]]]) - y = masked_rnn(x, mask=mask) - xx = torch.tensor([[[1.0]]]) - y = masked_rnn.step(xx) - y = masked_rnn.step(xx, mask=mask) \ No newline at end of file diff --git a/test/modules/test_other_modules.py b/test/modules/test_other_modules.py index 2645424e..c5462623 100644 --- a/test/modules/test_other_modules.py +++ b/test/modules/test_other_modules.py @@ -2,50 +2,14 @@ import unittest import torch -from fastNLP.modules.other_modules import GroupNorm, LayerNormalization, BiLinear, BiAffine +from fastNLP.modules.encoder.star_transformer import StarTransformer -class TestGroupNorm(unittest.TestCase): - def test_case_1(self): - gn = GroupNorm(num_features=1, num_groups=10, eps=1.5e-5) - x = torch.randn((20, 50, 10)) - y = gn(x) - - -class TestLayerNormalization(unittest.TestCase): - def test_case_1(self): - ln = LayerNormalization(layer_size=5, eps=2e-3) - x = torch.randn((20, 50, 5)) - y = ln(x) - - -class TestBiLinear(unittest.TestCase): - def test_case_1(self): - bl = BiLinear(n_left=5, n_right=5, n_out=10, bias=True) - x_left = torch.randn((7, 10, 20, 5)) - x_right = torch.randn((7, 10, 20, 5)) - y = bl(x_left, x_right) - print(bl) - bl2 = BiLinear(n_left=15, n_right=15, n_out=10, bias=True) - - -class TestBiAffine(unittest.TestCase): - def test_case_1(self): - batch_size = 16 - encoder_length = 21 - decoder_length = 32 - layer = BiAffine(10, 10, 25, biaffine=True) - decoder_input = torch.randn((batch_size, encoder_length, 10)) - encoder_input = torch.randn((batch_size, decoder_length, 10)) - y = layer(decoder_input, encoder_input) - self.assertEqual(tuple(y.shape), (batch_size, 25, encoder_length, decoder_length)) - - def test_case_2(self): - batch_size = 16 - encoder_length = 21 - decoder_length = 32 - layer = BiAffine(10, 10, 25, biaffine=False) - decoder_input = torch.randn((batch_size, encoder_length, 10)) - encoder_input = torch.randn((batch_size, decoder_length, 10)) - y = layer(decoder_input, encoder_input) - self.assertEqual(tuple(y.shape), (batch_size, 25, encoder_length, 1)) +class TestStarTransformer(unittest.TestCase): + def test_1(self): + model = StarTransformer(num_layers=6, hidden_size=100, num_head=8, head_dim=20, max_len=100) + x = torch.rand(16, 45, 100) + mask = torch.ones(16, 45).byte() + y, yn = model(x, mask) + self.assertEqual(tuple(y.size()), (16, 45, 100)) + self.assertEqual(tuple(yn.size()), (16, 100)) diff --git a/test/modules/test_utils.py b/test/modules/test_utils.py deleted file mode 100644 index 1d3cfcac..00000000 --- a/test/modules/test_utils.py +++ /dev/null @@ -1,9 +0,0 @@ -import unittest - - -class TestUtils(unittest.TestCase): - def test_case_1(self): - pass - - def test_case_2(self): - pass diff --git a/test/test_tutorial.py b/test/test_tutorial.py deleted file mode 100644 index 68cb6a41..00000000 --- a/test/test_tutorial.py +++ /dev/null @@ -1,91 +0,0 @@ -import unittest - -from fastNLP import DataSet -from fastNLP import Instance -from fastNLP import Tester -from fastNLP import Vocabulary -from fastNLP.core.losses import CrossEntropyLoss -from fastNLP.core.metrics import AccuracyMetric -from fastNLP.models import CNNText - - -class TestTutorial(unittest.TestCase): - def test_tutorial(self): - # 从csv读取数据到DataSet - sample_path = "test/data_for_tests/tutorial_sample_dataset.csv" - dataset = DataSet.read_csv(sample_path, headers=('raw_sentence', 'label'), - sep='\t') - print(len(dataset)) - print(dataset[0]) - - dataset.append(Instance(raw_sentence='fake data', label='0')) - dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence') - # label转int - dataset.apply(lambda x: int(x['label']), new_field_name='label') - - # 使用空格分割句子 - def split_sent(ins): - return ins['raw_sentence'].split() - - dataset.apply(split_sent, new_field_name='words') - # 增加长度信息 - dataset.apply(lambda x: len(x['words']), new_field_name='seq_len') - print(len(dataset)) - print(dataset[0]) - - # DataSet.drop(func)筛除数据 - dataset.drop(lambda x: x['seq_len'] <= 3) - print(len(dataset)) - - # 设置DataSet中，哪些field要转为tensor - # set target，loss或evaluate中的golden，计算loss，模型评估时使用 - dataset.set_target("label") - # set input，模型forward时使用 - dataset.set_input("words") - - # 分出测试集、训练集 - test_data, train_data = dataset.split(0.5) - print(len(test_data)) - print(len(train_data)) - - # 构建词表, Vocabulary.add(word) - vocab = Vocabulary(min_freq=2) - train_data.apply(lambda x: [vocab.add(word) for word in x['words']]) - vocab.build_vocab() - - # index句子, Vocabulary.to_index(word) - train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words') - test_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words') - print(test_data[0]) - - model = CNNText(embed_num=len(vocab), embed_dim=50, num_classes=5, padding=2, dropout=0.1) - - from fastNLP import Trainer - from copy import deepcopy - - # 更改DataSet中对应field的名称，要以模型的forward等参数名一致 - train_data.rename_field('words', 'word_seq') # input field 与 forward 参数一致 - train_data.rename_field('label', 'label_seq') - test_data.rename_field('words', 'word_seq') - test_data.rename_field('label', 'label_seq') - - # 实例化Trainer，传入模型和数据，进行训练 - copy_model = deepcopy(model) - overfit_trainer = Trainer(train_data=test_data, model=copy_model, - loss=CrossEntropyLoss(pred="output", target="label_seq"), - metrics=AccuracyMetric(pred="predict", target="label_seq"), n_epochs=10, batch_size=4, - dev_data=test_data, save_path="./save") - overfit_trainer.train() - - trainer = Trainer(train_data=train_data, model=model, - loss=CrossEntropyLoss(pred="output", target="label_seq"), - metrics=AccuracyMetric(pred="predict", target="label_seq"), n_epochs=10, batch_size=4, - dev_data=test_data, save_path="./save") - trainer.train() - print('Train finished!') - - # 使用fastNLP的Tester测试脚本 - tester = Tester(data=test_data, model=model, metrics=AccuracyMetric(pred="predict", target="label_seq"), - batch_size=4) - acc = tester.test() - print(acc) diff --git a/test/test_tutorials.py b/test/test_tutorials.py new file mode 100644 index 00000000..128e4235 --- /dev/null +++ b/test/test_tutorials.py @@ -0,0 +1,166 @@ +import unittest + +from fastNLP import DataSet +from fastNLP import Instance +from fastNLP import Vocabulary +from fastNLP.core.losses import CrossEntropyLoss +from fastNLP.core.metrics import AccuracyMetric + + +class TestTutorial(unittest.TestCase): + def test_fastnlp_10min_tutorial(self): + # 从csv读取数据到DataSet + sample_path = "test/data_for_tests/tutorial_sample_dataset.csv" + dataset = DataSet.read_csv(sample_path, headers=('raw_sentence', 'label'), + sep='\t') + print(len(dataset)) + print(dataset[0]) + print(dataset[-3]) + + dataset.append(Instance(raw_sentence='fake data', label='0')) + # 将所有数字转为小写 + dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence') + # label转int + dataset.apply(lambda x: int(x['label']), new_field_name='label') + + # 使用空格分割句子 + def split_sent(ins): + return ins['raw_sentence'].split() + + dataset.apply(split_sent, new_field_name='words') + + # 增加长度信息 + dataset.apply(lambda x: len(x['words']), new_field_name='seq_len') + print(len(dataset)) + print(dataset[0]) + + # DataSet.drop(func)筛除数据 + dataset.drop(lambda x: x['seq_len'] <= 3, inplace=True) + print(len(dataset)) + + # 设置DataSet中，哪些field要转为tensor + # set target，loss或evaluate中的golden，计算loss，模型评估时使用 + dataset.set_target("label") + # set input，模型forward时使用 + dataset.set_input("words", "seq_len") + + # 分出测试集、训练集 + test_data, train_data = dataset.split(0.5) + print(len(test_data)) + print(len(train_data)) + + # 构建词表, Vocabulary.add(word) + vocab = Vocabulary(min_freq=2) + train_data.apply(lambda x: [vocab.add(word) for word in x['words']]) + vocab.build_vocab() + + # index句子, Vocabulary.to_index(word) + train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words') + test_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words') + print(test_data[0]) + + # 如果你们需要做强化学习或者GAN之类的项目，你们也可以使用这些数据预处理的工具 + from fastNLP.core.batch import Batch + from fastNLP.core.sampler import RandomSampler + + batch_iterator = Batch(dataset=train_data, batch_size=2, sampler=RandomSampler()) + for batch_x, batch_y in batch_iterator: + print("batch_x has: ", batch_x) + print("batch_y has: ", batch_y) + break + + from fastNLP.models import CNNText + model = CNNText((len(vocab), 50), num_classes=5, padding=2, dropout=0.1) + + from fastNLP import Trainer + from copy import deepcopy + + # 更改DataSet中对应field的名称，要以模型的forward等参数名一致 + train_data.rename_field('label', 'label_seq') + test_data.rename_field('label', 'label_seq') + + loss = CrossEntropyLoss(pred="output", target="label_seq") + metric = AccuracyMetric(pred="predict", target="label_seq") + + # 实例化Trainer，传入模型和数据，进行训练 + # 先在test_data拟合（确保模型的实现是正确的） + copy_model = deepcopy(model) + overfit_trainer = Trainer(model=copy_model, train_data=test_data, dev_data=test_data, + loss=loss, + metrics=metric, + save_path=None, + batch_size=32, + n_epochs=5) + overfit_trainer.train() + + # 用train_data训练，在test_data验证 + trainer = Trainer(model=model, train_data=train_data, dev_data=test_data, + loss=CrossEntropyLoss(pred="output", target="label_seq"), + metrics=AccuracyMetric(pred="predict", target="label_seq"), + save_path=None, + batch_size=32, + n_epochs=5) + trainer.train() + print('Train finished!') + + # 调用Tester在test_data上评价效果 + from fastNLP import Tester + + tester = Tester(data=test_data, model=model, metrics=AccuracyMetric(pred="predict", target="label_seq"), + batch_size=4) + acc = tester.test() + print(acc) + + def test_fastnlp_1min_tutorial(self): + # tutorials/fastnlp_1min_tutorial.ipynb + data_path = "test/data_for_tests/tutorial_sample_dataset.csv" + ds = DataSet.read_csv(data_path, headers=('raw_sentence', 'label'), sep='\t') + print(ds[1]) + + # 将所有数字转为小写 + ds.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence') + # label转int + ds.apply(lambda x: int(x['label']), new_field_name='target', is_target=True) + + def split_sent(ins): + return ins['raw_sentence'].split() + + ds.apply(split_sent, new_field_name='words', is_input=True) + + # 分割训练集/验证集 + train_data, dev_data = ds.split(0.3) + print("Train size: ", len(train_data)) + print("Test size: ", len(dev_data)) + + from fastNLP import Vocabulary + vocab = Vocabulary(min_freq=2) + train_data.apply(lambda x: [vocab.add(word) for word in x['words']]) + + # index句子, Vocabulary.to_index(word) + train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words', + is_input=True) + dev_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words', + is_input=True) + + from fastNLP.models import CNNText + model = CNNText((len(vocab), 50), num_classes=5, padding=2, dropout=0.1) + + from fastNLP import Trainer, CrossEntropyLoss, AccuracyMetric, Adam + + trainer = Trainer(model=model, + train_data=train_data, + dev_data=dev_data, + loss=CrossEntropyLoss(), + optimizer= Adam(), + metrics=AccuracyMetric(target='target') + ) + trainer.train() + print('Train finished!') + + def setUp(self): + import os + self._init_wd = os.path.abspath(os.curdir) + + def tearDown(self): + import os + os.chdir(self._init_wd) diff --git a/tutorials/README.md b/tutorials/README.md index 1de342e6..83df2bb9 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -1,12 +1,7 @@ # fastNLP 教程 ### 上手教程 Quick Start -- 一分钟上手：`fastnlp_1min_tutorial.ipynb` [Click Here](https://github.com/fastnlp/fastNLP/tree/master/tutorials/fastnlp_1min_tutorial.ipynb) -- 十分钟上手：`fastnlp_10min_tutorial.ipynb` [Click Here](https://github.com/fastnlp/fastNLP/tree/master/tutorials/fastnlp_10min_tutorial.ipynb) +`quickstart.ipynb` [Click Here](https://github.com/fastnlp/fastNLP/tree/master/tutorials/quickstart.ipynb) -### 进阶教程 Advanced Tutorial -- `fastnlp_advanced_tutorial/advance_tutorial.ipynb` [Click Here](https://github.com/fastnlp/fastNLP/tree/master/tutorials/fastnlp_advanced_tutorial/advance_tutorial.ipynb) - - -### 开发者指南 Developer Guide -- `tutorial_for_developer.md` [Click Here](https://github.com/fastnlp/fastNLP/tree/master/tutorials/tutorial_for_developer.md) +### 详细教程 Tutorial 1 +十分钟上手：`tutorial_1.ipynb` [Click Here](https://github.com/fastnlp/fastNLP/tree/master/tutorials/tutorial_1.ipynb) diff --git a/tutorials/fastnlp_10min_tutorial.ipynb b/tutorials/fastnlp_10min_tutorial.ipynb deleted file mode 100644 index 534c4e49..00000000 --- a/tutorials/fastnlp_10min_tutorial.ipynb +++ /dev/null @@ -1,762 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "fastNLP10 分钟上手教程\n", - "-------\n", - "\n", - "fastNLP提供方便的数据预处理，训练和测试模型的功能" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "如果您还没有通过pip安装fastNLP，可以执行下面的操作加载当前模块" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "sys.path.append(\"../\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "DataSet & Instance\n", - "------\n", - "\n", - "fastNLP用DataSet和Instance保存和处理数据。每个DataSet表示一个数据集，每个Instance表示一个数据样本。一个DataSet存有多个Instance，每个Instance可以自定义存哪些内容。\n", - "\n", - "有一些read_*方法，可以轻松从文件读取数据，存成DataSet。" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "77\n" - ] - } - ], - "source": [ - "from fastNLP import DataSet\n", - "from fastNLP import Instance\n", - "\n", - "# 从csv读取数据到DataSet\n", - "dataset = DataSet.read_csv('sample_data/tutorial_sample_dataset.csv', headers=('raw_sentence', 'label'), sep='\\t')\n", - "print(len(dataset))" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=str}\n", - "{'raw_sentence': The plot is romantic comedy boilerplate from start to finish . type=str,\n", - "'label': 2 type=str}\n" - ] - } - ], - "source": [ - "# 使用数字索引[k]，获取第k个样本\n", - "print(dataset[0])\n", - "\n", - "# 索引也可以是负数\n", - "print(dataset[-3])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Instance\n", - "Instance表示一个样本，由一个或多个field（域，属性，特征）组成，每个field有名字和值。\n", - "\n", - "在初始化Instance时即可定义它包含的域，使用 \"field_name=field_value\"的写法。" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'raw_sentence': fake data type=str,\n", - "'label': 0 type=str}" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# DataSet.append(Instance)加入新数据\n", - "dataset.append(Instance(raw_sentence='fake data', label='0'))\n", - "dataset[-1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## DataSet.apply方法\n", - "数据预处理利器" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'raw_sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=str}\n" - ] - } - ], - "source": [ - "# 将所有数字转为小写\n", - "dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence')\n", - "print(dataset[0])" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'raw_sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=int}\n" - ] - } - ], - "source": [ - "# label转int\n", - "dataset.apply(lambda x: int(x['label']), new_field_name='label')\n", - "print(dataset[0])" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'raw_sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=int,\n", - "'words': ['a', 'series', 'of', 'escapades', 'demonstrating', 'the', 'adage', 'that', 'what', 'is', 'good', 'for', 'the', 'goose', 'is', 'also', 'good', 'for', 'the', 'gander', ',', 'some', 'of', 'which', 'occasionally', 'amuses', 'but', 'none', 'of', 'which', 'amounts', 'to', 'much', 'of', 'a', 'story', '.'] type=list}\n" - ] - } - ], - "source": [ - "# 使用空格分割句子\n", - "def split_sent(ins):\n", - " return ins['raw_sentence'].split()\n", - "dataset.apply(split_sent, new_field_name='words')\n", - "print(dataset[0])" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'raw_sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=int,\n", - "'words': ['a', 'series', 'of', 'escapades', 'demonstrating', 'the', 'adage', 'that', 'what', 'is', 'good', 'for', 'the', 'goose', 'is', 'also', 'good', 'for', 'the', 'gander', ',', 'some', 'of', 'which', 'occasionally', 'amuses', 'but', 'none', 'of', 'which', 'amounts', 'to', 'much', 'of', 'a', 'story', '.'] type=list,\n", - "'seq_len': 37 type=int}\n" - ] - } - ], - "source": [ - "# 增加长度信息\n", - "dataset.apply(lambda x: len(x['words']), new_field_name='seq_len')\n", - "print(dataset[0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## DataSet.drop\n", - "筛选数据" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "77\n" - ] - } - ], - "source": [ - "# 删除低于某个长度的词语\n", - "dataset.drop(lambda x: x['seq_len'] <= 3)\n", - "print(len(dataset))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 配置DataSet\n", - "1. 哪些域是特征，哪些域是标签\n", - "2. 切分训练集/验证集" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [], - "source": [ - "# 设置DataSet中，哪些field要转为tensor\n", - "\n", - "# set target，loss或evaluate中的golden，计算loss，模型评估时使用\n", - "dataset.set_target(\"label\")\n", - "# set input，模型forward时使用\n", - "dataset.set_input(\"words\")" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "54\n", - "23\n" - ] - } - ], - "source": [ - "# 分出测试集、训练集\n", - "\n", - "test_data, train_data = dataset.split(0.3)\n", - "print(len(test_data))\n", - "print(len(train_data))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Vocabulary\n", - "------\n", - "\n", - "fastNLP中的Vocabulary轻松构建词表，将词转成数字" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'raw_sentence': a welcome relief from baseball movies that try too hard to be mythic , this one is a sweet and modest and ultimately winning story . type=str,\n", - "'label': 3 type=int,\n", - "'words': [4, 1, 1, 18, 1, 1, 13, 1, 1, 1, 8, 26, 1, 5, 35, 1, 11, 4, 1, 10, 1, 10, 1, 1, 1, 2] type=list,\n", - "'seq_len': 26 type=int}\n" - ] - } - ], - "source": [ - "from fastNLP import Vocabulary\n", - "\n", - "# 构建词表, Vocabulary.add(word)\n", - "vocab = Vocabulary(min_freq=2)\n", - "train_data.apply(lambda x: [vocab.add(word) for word in x['words']])\n", - "vocab.build_vocab()\n", - "\n", - "# index句子, Vocabulary.to_index(word)\n", - "train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words')\n", - "test_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='words')\n", - "\n", - "\n", - "print(test_data[0])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# 如果你们需要做强化学习或者GAN之类的项目，你们也可以使用这些数据预处理的工具\n", - "from fastNLP.core.batch import Batch\n", - "from fastNLP.core.sampler import RandomSampler\n", - "\n", - "batch_iterator = Batch(dataset=train_data, batch_size=2, sampler=RandomSampler())\n", - "for batch_x, batch_y in batch_iterator:\n", - " print(\"batch_x has: \", batch_x)\n", - " print(\"batch_y has: \", batch_y)\n", - " break" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Model\n", - "定义一个PyTorch模型" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "CNNText(\n", - " (embed): Embedding(\n", - " (embed): Embedding(59, 50, padding_idx=0)\n", - " (dropout): Dropout(p=0.0)\n", - " )\n", - " (conv_pool): ConvMaxpool(\n", - " (convs): ModuleList(\n", - " (0): Conv1d(50, 3, kernel_size=(3,), stride=(1,), padding=(2,))\n", - " (1): Conv1d(50, 4, kernel_size=(4,), stride=(1,), padding=(2,))\n", - " (2): Conv1d(50, 5, kernel_size=(5,), stride=(1,), padding=(2,))\n", - " )\n", - " )\n", - " (dropout): Dropout(p=0.1)\n", - " (fc): Linear(\n", - " (linear): Linear(in_features=12, out_features=5, bias=True)\n", - " )\n", - ")" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from fastNLP.models import CNNText\n", - "model = CNNText(embed_num=len(vocab), embed_dim=50, num_classes=5, padding=2, dropout=0.1)\n", - "model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "这是上述模型的forward方法。如果你不知道什么是forward方法，请参考我们的PyTorch教程。\n", - "\n", - "注意两点：\n", - "1. forward参数名字叫**word_seq**，请记住。\n", - "2. forward的返回值是一个**dict**，其中有个key的名字叫**output**。\n", - "\n", - "```Python\n", - " def forward(self, word_seq):\n", - " \"\"\"\n", - "\n", - " :param word_seq: torch.LongTensor, [batch_size, seq_len]\n", - " :return output: dict of torch.LongTensor, [batch_size, num_classes]\n", - " \"\"\"\n", - " x = self.embed(word_seq) # [N,L] -> [N,L,C]\n", - " x = self.conv_pool(x) # [N,L,C] -> [N,C]\n", - " x = self.dropout(x)\n", - " x = self.fc(x) # [N,C] -> [N, N_class]\n", - " return {'output': x}\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "这是上述模型的predict方法，是用来直接输出该任务的预测结果，与forward目的不同。\n", - "\n", - "注意两点：\n", - "1. predict参数名也叫**word_seq**。\n", - "2. predict的返回值是也一个**dict**，其中有个key的名字叫**predict**。\n", - "\n", - "```\n", - " def predict(self, word_seq):\n", - " \"\"\"\n", - "\n", - " :param word_seq: torch.LongTensor, [batch_size, seq_len]\n", - " :return predict: dict of torch.LongTensor, [batch_size, seq_len]\n", - " \"\"\"\n", - " output = self(word_seq)\n", - " _, predict = output['output'].max(dim=1)\n", - " return {'predict': predict}\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Trainer & Tester\n", - "------\n", - "\n", - "使用fastNLP的Trainer训练模型" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [], - "source": [ - "from fastNLP import Trainer\n", - "from copy import deepcopy\n", - "from fastNLP.core.losses import CrossEntropyLoss\n", - "from fastNLP.core.metrics import AccuracyMetric\n", - "\n", - "\n", - "# 更改DataSet中对应field的名称，与模型的forward的参数名一致\n", - "# 因为forward的参数叫word_seq, 所以要把原本叫words的field改名为word_seq\n", - "# 这里的演示是让你了解这种**命名规则**\n", - "train_data.rename_field('words', 'word_seq')\n", - "test_data.rename_field('words', 'word_seq')\n", - "\n", - "# 顺便把label换名为label_seq\n", - "train_data.rename_field('label', 'label_seq')\n", - "test_data.rename_field('label', 'label_seq')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### loss\n", - "训练模型需要提供一个损失函数\n", - "\n", - "下面提供了一个在分类问题中常用的交叉熵损失。注意它的**初始化参数**。\n", - "\n", - "pred参数对应的是模型的forward返回的dict的一个key的名字，这里是\"output\"。\n", - "\n", - "target参数对应的是dataset作为标签的field的名字，这里是\"label_seq\"。" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [], - "source": [ - "loss = CrossEntropyLoss(pred=\"output\", target=\"label_seq\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Metric\n", - "定义评价指标\n", - "\n", - "这里使用准确率。参数的“命名规则”跟上面类似。\n", - "\n", - "pred参数对应的是模型的predict方法返回的dict的一个key的名字，这里是\"predict\"。\n", - "\n", - "target参数对应的是dataset作为标签的field的名字，这里是\"label_seq\"。" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [], - "source": [ - "metric = AccuracyMetric(pred=\"predict\", target=\"label_seq\")" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "input fields after batch(if batch size is 2):\n", - "\tword_seq: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26]) \n", - "target fields after batch(if batch size is 2):\n", - "\tlabel_seq: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", - "\n", - "training epochs started 2019-01-12 17-07-51\n" - ] - }, - { - "data": { - "text/plain": [ - "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=10), HTML(value='')), layout=Layout(display='…" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation at Epoch 1/5. Step:2/10. AccuracyMetric: acc=0.425926\n", - "Evaluation at Epoch 2/5. Step:4/10. AccuracyMetric: acc=0.425926\n", - "Evaluation at Epoch 3/5. Step:6/10. AccuracyMetric: acc=0.611111\n", - "Evaluation at Epoch 4/5. Step:8/10. AccuracyMetric: acc=0.648148\n", - "Evaluation at Epoch 5/5. Step:10/10. AccuracyMetric: acc=0.703704\n", - "\n", - "In Epoch:5/Step:10, got best dev performance:AccuracyMetric: acc=0.703704\n", - "Reloaded the best model.\n" - ] - }, - { - "data": { - "text/plain": [ - "{'best_eval': {'AccuracyMetric': {'acc': 0.703704}},\n", - " 'best_epoch': 5,\n", - " 'best_step': 10,\n", - " 'seconds': 0.62}" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 实例化Trainer，传入模型和数据，进行训练\n", - "# 先在test_data拟合（确保模型的实现是正确的）\n", - "copy_model = deepcopy(model)\n", - "overfit_trainer = Trainer(model=copy_model, train_data=test_data, dev_data=test_data,\n", - " loss=loss,\n", - " metrics=metric,\n", - " save_path=None,\n", - " batch_size=32,\n", - " n_epochs=5)\n", - "overfit_trainer.train()" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "input fields after batch(if batch size is 2):\n", - "\tword_seq: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 20]) \n", - "target fields after batch(if batch size is 2):\n", - "\tlabel_seq: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", - "\n", - "training epochs started 2019-01-12 17-09-05\n" - ] - }, - { - "data": { - "text/plain": [ - "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=5), HTML(value='')), layout=Layout(display='i…" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation at Epoch 1/5. Step:1/5. AccuracyMetric: acc=0.37037\n", - "Evaluation at Epoch 2/5. Step:2/5. AccuracyMetric: acc=0.37037\n", - "Evaluation at Epoch 3/5. Step:3/5. AccuracyMetric: acc=0.462963\n", - "Evaluation at Epoch 4/5. Step:4/5. AccuracyMetric: acc=0.425926\n", - "Evaluation at Epoch 5/5. Step:5/5. AccuracyMetric: acc=0.481481\n", - "\n", - "In Epoch:5/Step:5, got best dev performance:AccuracyMetric: acc=0.481481\n", - "Reloaded the best model.\n", - "Train finished!\n" - ] - } - ], - "source": [ - "# 用train_data训练，在test_data验证\n", - "trainer = Trainer(model=model, train_data=train_data, dev_data=test_data,\n", - " loss=CrossEntropyLoss(pred=\"output\", target=\"label_seq\"),\n", - " metrics=AccuracyMetric(pred=\"predict\", target=\"label_seq\"),\n", - " save_path=None,\n", - " batch_size=32,\n", - " n_epochs=5)\n", - "trainer.train()\n", - "print('Train finished!')" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[tester] \n", - "AccuracyMetric: acc=0.481481\n", - "{'AccuracyMetric': {'acc': 0.481481}}\n" - ] - } - ], - "source": [ - "# 调用Tester在test_data上评价效果\n", - "from fastNLP import Tester\n", - "\n", - "tester = Tester(data=test_data, model=model, metrics=AccuracyMetric(pred=\"predict\", target=\"label_seq\"),\n", - " batch_size=4)\n", - "acc = tester.test()\n", - "print(acc)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# In summary\n", - "\n", - "## fastNLP Trainer的伪代码逻辑\n", - "### 1. 准备DataSet，假设DataSet中共有如下的fields\n", - " ['raw_sentence', 'word_seq1', 'word_seq2', 'raw_label','label']\n", - " 通过\n", - " DataSet.set_input('word_seq1', word_seq2', flag=True)将'word_seq1', 'word_seq2'设置为input\n", - " 通过\n", - " DataSet.set_target('label', flag=True)将'label'设置为target\n", - "### 2. 初始化模型\n", - " class Model(nn.Module):\n", - " def __init__(self):\n", - " xxx\n", - " def forward(self, word_seq1, word_seq2):\n", - " # (1) 这里使用的形参名必须和DataSet中的input field的名称对应。因为我们是通过形参名, 进行赋值的\n", - " # (2) input field的数量可以多于这里的形参数量。但是不能少于。\n", - " xxxx\n", - " # 输出必须是一个dict\n", - "### 3. Trainer的训练过程\n", - " (1) 从DataSet中按照batch_size取出一个batch，调用Model.forward\n", - " (2) 将 Model.forward的结果 与 标记为target的field 传入Losser当中。\n", - " 由于每个人写的Model.forward的output的dict可能key并不一样，比如有人是{'pred':xxx}, {'output': xxx}; \n", - " 另外每个人将target可能也会设置为不同的名称, 比如有人是label, 有人设置为target；\n", - " 为了解决以上的问题，我们的loss提供映射机制\n", - " 比如CrossEntropyLosser的需要的输入是(prediction, target)。但是forward的output是{'output': xxx}; 'label'是target\n", - " 那么初始化losser的时候写为CrossEntropyLosser(prediction='output', target='label')即可\n", - " (3) 对于Metric是同理的\n", - " Metric计算也是从 forward的结果中取值 与 设置target的field中取值。 也是可以通过映射找到对应的值 \n", - " \n", - " \n", - "\n", - "## 一些问题.\n", - "### 1. DataSet中为什么需要设置input和target\n", - " 只有被设置为input或者target的数据才会在train的过程中被取出来\n", - " (1.1) 我们只会在设置为input的field中寻找传递给Model.forward的参数。\n", - " (1.2) 我们在传递值给losser或者metric的时候会使用来自: \n", - " (a)Model.forward的output\n", - " (b)被设置为target的field\n", - " \n", - "\n", - "### 2. 我们是通过forwad中的形参名将DataSet中的field赋值给对应的参数\n", - " (1.1) 构建模型过程中，\n", - " 例如:\n", - " DataSet中x，seq_lens是input，那么forward就应该是\n", - " def forward(self, x, seq_lens):\n", - " pass\n", - " 我们是通过形参名称进行匹配的field的\n", - " \n", - "\n", - "\n", - "### 1. 加载数据到DataSet\n", - "### 2. 使用apply操作对DataSet进行预处理\n", - " (2.1) 处理过程中将某些field设置为input，某些field设置为target\n", - "### 3. 构建模型\n", - " (3.1) 构建模型过程中，需要注意forward函数的形参名需要和DataSet中设置为input的field名称是一致的。\n", - " 例如:\n", - " DataSet中x，seq_lens是input，那么forward就应该是\n", - " def forward(self, x, seq_lens):\n", - " pass\n", - " 我们是通过形参名称进行匹配的field的\n", - " (3.2) 模型的forward的output需要是dict类型的。\n", - " 建议将输出设置为{\"pred\": xx}.\n", - " \n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.7" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/tutorials/fastnlp_1min_tutorial.ipynb b/tutorials/fastnlp_1min_tutorial.ipynb deleted file mode 100644 index 7a35d992..00000000 --- a/tutorials/fastnlp_1min_tutorial.ipynb +++ /dev/null @@ -1,248 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "# fastNLP 1分钟上手教程" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## step 1\n", - "读取数据集" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "c:\\users\\zyfeng\\miniconda3\\envs\\fastnlp\\lib\\site-packages\\tqdm\\autonotebook\\__init__.py:14: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n", - " \" (e.g. in jupyter console)\", TqdmExperimentalWarning)\n" - ] - } - ], - "source": [ - "import sys\n", - "sys.path.append(\"../\")\n", - "\n", - "from fastNLP import DataSet\n", - "\n", - "data_path = \"./sample_data/tutorial_sample_dataset.csv\"\n", - "ds = DataSet.read_csv(data_path, headers=('raw_sentence', 'label'), sep='\\t')" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'raw_sentence': This quiet , introspective and entertaining independent is worth seeking . type=str,\n", - "'label': 4 type=str}" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "ds[1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## step 2\n", - "数据预处理\n", - "1. 类型转换\n", - "2. 切分验证集\n", - "3. 构建词典" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "# 将所有数字转为小写\n", - "ds.apply(lambda x: x['raw_sentence'].lower(), new_field_name='raw_sentence')\n", - "# label转int\n", - "ds.apply(lambda x: int(x['label']), new_field_name='label_seq', is_target=True)\n", - "\n", - "def split_sent(ins):\n", - " return ins['raw_sentence'].split()\n", - "ds.apply(split_sent, new_field_name='words', is_input=True)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Train size: 54\n", - "Test size: 23\n" - ] - } - ], - "source": [ - "# 分割训练集/验证集\n", - "train_data, dev_data = ds.split(0.3)\n", - "print(\"Train size: \", len(train_data))\n", - "print(\"Test size: \", len(dev_data))" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "from fastNLP import Vocabulary\n", - "vocab = Vocabulary(min_freq=2)\n", - "train_data.apply(lambda x: [vocab.add(word) for word in x['words']])\n", - "\n", - "# index句子, Vocabulary.to_index(word)\n", - "train_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='word_seq', is_input=True)\n", - "dev_data.apply(lambda x: [vocab.to_index(word) for word in x['words']], new_field_name='word_seq', is_input=True)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## step 3\n", - " 定义模型" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "from fastNLP.models import CNNText\n", - "model = CNNText(embed_num=len(vocab), embed_dim=50, num_classes=5, padding=2, dropout=0.1)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## step 4\n", - "开始训练" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "input fields after batch(if batch size is 2):\n", - "\twords: (1)type:numpy.ndarray (2)dtype:object, (3)shape:(2,) \n", - "\tword_seq: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 25]) \n", - "target fields after batch(if batch size is 2):\n", - "\tlabel_seq: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", - "\n", - "training epochs started 2019-01-12 17-00-48\n" - ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "23979df0f63e446fbb0406b919b91dd3", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=6), HTML(value='')), layout=Layout(display='i…" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation at Epoch 1/3. Step:2/6. AccuracyMetric: acc=0.173913\n", - "Evaluation at Epoch 2/3. Step:4/6. AccuracyMetric: acc=0.26087\n", - "Evaluation at Epoch 3/3. Step:6/6. AccuracyMetric: acc=0.304348\n", - "\n", - "In Epoch:3/Step:6, got best dev performance:AccuracyMetric: acc=0.304348\n", - "Reloaded the best model.\n", - "Train finished!\n" - ] - } - ], - "source": [ - "from fastNLP import Trainer, CrossEntropyLoss, AccuracyMetric\n", - "trainer = Trainer(model=model, \n", - " train_data=train_data, \n", - " dev_data=dev_data,\n", - " loss=CrossEntropyLoss(),\n", - " metrics=AccuracyMetric()\n", - " )\n", - "trainer.train()\n", - "print('Train finished!')\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 本教程结束。更多操作请参考进阶教程。" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.7" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/tutorials/fastnlp_advanced_tutorial/advance_tutorial.ipynb b/tutorials/fastnlp_advanced_tutorial/advance_tutorial.ipynb deleted file mode 100644 index a787eeaf..00000000 --- a/tutorials/fastnlp_advanced_tutorial/advance_tutorial.ipynb +++ /dev/null @@ -1,1169 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# fastNLP开发进阶教程\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 组织数据部分\n", - "## DataSet & Instance\n", - "fastNLP用DataSet和Instance保存和处理数据。每个DataSet表示一个数据集，每个Instance表示一个数据样本。一个DataSet存有多个Instance，每个Instance可以自定义存哪些内容。" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/remote-home/ygxu/anaconda3/envs/no-fastnlp/lib/python3.7/site-packages/tqdm/autonotebook/__init__.py:14: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n", - " \" (e.g. in jupyter console)\", TqdmExperimentalWarning)\n" - ] - } - ], - "source": [ - "# 声明部件\n", - "import torch\n", - "import fastNLP\n", - "from fastNLP import DataSet\n", - "from fastNLP import Instance\n", - "from fastNLP import Vocabulary\n", - "from fastNLP import Trainer\n", - "from fastNLP import Tester" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Instance\n", - "Instance表示一个样本，由一个或者多个field（域、属性、特征）组成，每个field具有自己的名字以及值\n", - "在初始化Instance的时候可以定义它包含的field，使用\"field_name=field_value\"的写法" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'premise': an premise example . type=str,\n", - "'hypothesis': an hypothesis example. type=str,\n", - "'label': 1 type=int}" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 组织一个Instance，这个Instance由premise、hypothesis、label三个field组成\n", - "instance = Instance(premise='an premise example .', hypothesis='an hypothesis example.', label=1)\n", - "instance" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'premise': an premise example . type=str,\n", - "'hypothesis': an hypothesis example. type=str,\n", - "'label': 1 type=int},\n", - "{'premise': an premise example . type=str,\n", - "'hypothesis': an hypothesis example. type=str,\n", - "'label': 1 type=int})" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "data_set = DataSet([instance] * 5)\n", - "data_set.append(instance)\n", - "data_set[-2: ]" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'premise': an premise example . type=str,\n", - "'hypothesis': an hypothesis example. type=str,\n", - "'label': 1 type=int},\n", - "{'premise': the second premise example . type=str,\n", - "'hypothesis': the second hypothesis example. type=str,\n", - "'label': 1 type=str})" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 如果某一个field的类型与dataset对应的field类型不一样仍可被加入dataset中\n", - "instance2 = Instance(premise='the second premise example .', hypothesis='the second hypothesis example.', label='1')\n", - "try:\n", - " data_set.append(instance2)\n", - "except:\n", - " pass\n", - "data_set[-2: ]" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "cannot append instance\n" - ] - }, - { - "data": { - "text/plain": [ - "DataSet({'premise': an premise example . type=str,\n", - "'hypothesis': an hypothesis example. type=str,\n", - "'label': 1 type=int},\n", - "{'premise': the second premise example . type=str,\n", - "'hypothesis': the second hypothesis example. type=str,\n", - "'label': 1 type=str})" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 如果某一个field的名字不对，则该instance不能被append到dataset中\n", - "instance3 = Instance(premises='the third premise example .', hypothesis='the third hypothesis example.', label=1)\n", - "try:\n", - " data_set.append(instance3)\n", - "except:\n", - " print('cannot append instance')\n", - " pass\n", - "data_set[-2: ]" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'image': tensor([[ 2.1747, -1.0147, -1.3853, 0.0216, -0.4957],\n", - " [ 0.8138, -0.2933, -0.1217, -0.6027, 0.3932],\n", - " [ 0.6750, -1.1136, -1.3371, -0.0185, -0.3206],\n", - " [-0.5076, -0.3822, 0.1719, -0.6447, -0.5702],\n", - " [ 0.3804, 0.0889, 0.8027, -0.7121, -0.7320]]) type=torch.Tensor,\n", - "'label': 0 type=int})" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 除了文本以外，还可以将tensor作为其中一个field的value\n", - "import torch\n", - "tensor_ins = Instance(image=torch.randn(5, 5), label=0)\n", - "ds = DataSet()\n", - "ds.append(tensor_ins)\n", - "ds" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### DataSet\n", - "### 使用现有代码读取并组织DataSet\n", - "在DataSet类当中有一些read_* 方法，可以从文件中读取数据并组织DataSet" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "77" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import fastNLP\n", - "from fastNLP import DataSet\n", - "from fastNLP import Instance\n", - "\n", - "# 从csv读取数据到DataSet\n", - "# 类csv文件，即每一行为一个example的文件，都可以使用这种方法进行数据读取\n", - "dataset = DataSet.read_csv('tutorial_sample_dataset.csv', headers=('raw_sentence', 'label'), sep='\\t')\n", - "# 查看DataSet的大小\n", - "len(dataset)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=str}" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 使用数字索引[k]，获取第k个样本\n", - "dataset[0]" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "fastNLP.core.instance.Instance" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 获取的样本是一个Instance\n", - "type(dataset[0])" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", - "'label': 1 type=str},\n", - "{'raw_sentence': This quiet , introspective and entertaining independent is worth seeking . type=str,\n", - "'label': 4 type=str},\n", - "{'raw_sentence': Even fans of Ismail Merchant 's work , I suspect , would have a hard time sitting through this one . type=str,\n", - "'label': 1 type=str})" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 使用数字索引[a: b]，获取第a到第b个样本\n", - "dataset[0: 3]" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'raw_sentence': A film that clearly means to preach exclusively to the converted . type=str,\n", - "'label': 2 type=str}" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 索引也可以是负数\n", - "dataset[-1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 自行读取并组织DataSet\n", - "以SNLI数据集为例，\n", - "SNLI数据集的训练、验证、测试集分别三个文件组成：第一个文件每一行是一句话，代表一个example当中的premise；第二个文件每一行也是一句话，代表一个example当中的hypothesis；第三个文件每一行是一个label" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'premise': A person on a horse jumps over a broken down airplane . type=str,\n", - "'hypothesis': A person is training his horse for a competition . type=str,\n", - "'truth': 1\n", - " type=str}" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "data_path = ['premise', 'hypothesis', 'label']\n", - "\n", - "# 读入文件\n", - "with open(data_path[0]) as f:\n", - " premise = f.readlines()\n", - "\n", - "with open(data_path[1]) as f:\n", - " hypothesis = f.readlines()\n", - "\n", - "with open(data_path[2]) as f:\n", - " label = f.readlines()\n", - "\n", - "assert len(premise) == len(hypothesis) and len(hypothesis) == len(label)\n", - "\n", - "# 组织DataSet\n", - "data_set = DataSet()\n", - "for p, h, l in zip(premise, hypothesis, label):\n", - " p = p.strip() # 将行末空格去除\n", - " h = h.strip() # 将行末空格去除\n", - " data_set.append(Instance(premise=p, hypothesis=h, truth=l))\n", - "\n", - "data_set[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### DataSet的其他操作\n", - "在构建完毕DataSet后，仍然可以对DataSet的内容进行操作，函数接口为DataSet.apply()" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'premise': a woman is walking across the street eating a banana , while a man is following with his briefcase . type=str,\n", - "'hypothesis': An actress and her favorite assistant talk a walk in the city . type=str,\n", - "'truth': 1\n", - " type=str},\n", - "{'premise': a woman is walking across the street eating a banana , while a man is following with his briefcase . type=str,\n", - "'hypothesis': a woman eating a banana crosses a street type=str,\n", - "'truth': 0\n", - " type=str})" - ] - }, - "execution_count": 13, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 将premise域的所有文本转成小写\n", - "data_set.apply(lambda x: x['premise'].lower(), new_field_name='premise')\n", - "data_set[-2: ]" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'premise': a woman is walking across the street eating a banana , while a man is following with his briefcase . type=str,\n", - "'hypothesis': An actress and her favorite assistant talk a walk in the city . type=str,\n", - "'truth': 1 type=int},\n", - "{'premise': a woman is walking across the street eating a banana , while a man is following with his briefcase . type=str,\n", - "'hypothesis': a woman eating a banana crosses a street type=str,\n", - "'truth': 0 type=int})" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# label转int\n", - "data_set.apply(lambda x: int(x['truth']), new_field_name='truth')\n", - "data_set[-2: ]" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "DataSet({'premise': ['a', 'woman', 'is', 'walking', 'across', 'the', 'street', 'eating', 'a', 'banana', ',', 'while', 'a', 'man', 'is', 'following', 'with', 'his', 'briefcase', '.'] type=list,\n", - "'hypothesis': ['An', 'actress', 'and', 'her', 'favorite', 'assistant', 'talk', 'a', 'walk', 'in', 'the', 'city', '.'] type=list,\n", - "'truth': 1 type=int},\n", - "{'premise': ['a', 'woman', 'is', 'walking', 'across', 'the', 'street', 'eating', 'a', 'banana', ',', 'while', 'a', 'man', 'is', 'following', 'with', 'his', 'briefcase', '.'] type=list,\n", - "'hypothesis': ['a', 'woman', 'eating', 'a', 'banana', 'crosses', 'a', 'street'] type=list,\n", - "'truth': 0 type=int})" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 使用空格分割句子\n", - "def split_sent(ins):\n", - " return ins['premise'].split()\n", - "data_set.apply(split_sent, new_field_name='premise')\n", - "data_set.apply(lambda x: x['hypothesis'].split(), new_field_name='hypothesis')\n", - "data_set[-2:]" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(100, 97)" - ] - }, - "execution_count": 16, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 筛选数据\n", - "origin_data_set_len = len(data_set)\n", - "data_set.drop(lambda x: len(x['premise']) <= 6)\n", - "origin_data_set_len, len(data_set)" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'premise': ['a', 'woman', 'is', 'walking', 'across', 'the', 'street', 'eating', 'a', 'banana', ',', 'while', 'a', 'man', 'is', 'following', 'with', 'his', 'briefcase', '.'] type=list,\n", - "'hypothesis': ['a', 'woman', 'eating', 'a', 'banana', 'crosses', 'a', 'street'] type=list,\n", - "'truth': 0 type=int,\n", - "'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - "'hypothesis_len': [1, 1, 1, 1, 1, 1, 1, 1] type=list}" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 增加长度信息\n", - "data_set.apply(lambda x: [1] * len(x['premise']), new_field_name='premise_len')\n", - "data_set.apply(lambda x: [1] * len(x['hypothesis']), new_field_name='hypothesis_len')\n", - "data_set[-1]" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [], - "source": [ - "# 设定特征域、标签域\n", - "data_set.set_input(\"premise\", \"premise_len\", \"hypothesis\", \"hypothesis_len\")\n", - "data_set.set_target(\"truth\")" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'premise': ['a', 'woman', 'is', 'walking', 'across', 'the', 'street', 'eating', 'a', 'banana', ',', 'while', 'a', 'man', 'is', 'following', 'with', 'his', 'briefcase', '.'] type=list,\n", - "'hypothesis': ['a', 'woman', 'eating', 'a', 'banana', 'crosses', 'a', 'street'] type=list,\n", - "'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - "'hypothesis_len': [1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - "'label': 0 type=int}" - ] - }, - "execution_count": 19, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 重命名field\n", - "data_set.rename_field('truth', 'label')\n", - "data_set[-1]" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(49, 29, 19)" - ] - }, - "execution_count": 20, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 切分训练、验证集、测试集\n", - "train_data, vad_data = data_set.split(0.5)\n", - "dev_data, test_data = vad_data.split(0.4)\n", - "len(train_data), len(dev_data), len(test_data)" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [], - "source": [ - "# 深拷贝一个数据集\n", - "import copy\n", - "train_data_2, dev_data_2 = copy.deepcopy(train_data), copy.deepcopy(dev_data)\n", - "del copy" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### DataSet的总结：\n", - "将DataSet的ield设置为input和target后，这些field在接下来的代码中将被使用。其中被设置为input的field会被传递给Model.forward，这个过程是通过键匹配的方式进行的。举例如下： \n", - "假设DataSet中有'x1', 'x2', 'x3'被设置为了input，而 \n", - "   (1)函数是Model.forward(self, x1, x3), 那么DataSet中'x1', 'x3'会被传递给forward函数。多余的'x2'会被忽略 \n", - "   (2)函数是Model.forward(self, x1, x4), 这里多需要了一个'x4', 但是DataSet的input field中没有这个field，会报错。 \n", - "   (3)函数是Model.forward(self, x1, kwargs), 会把'x1', 'x2', 'x3'都传入。但如果是Model.forward(self, x4, kwargs)就会发生报错，因为没有'x4'。 \n", - "   (4)对于设置为target的field的名称，我们建议取名为'target'（如果只有一个需要predict的值），但是不强制。如果这个名称不是target，那么在加载loss函数的时候需要手动添加名称转换map" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Vocabulary\n", - "fastNLP中的Vocabulary轻松构建词表，并将词转成数字。构建词表有两种方式：根据数据集构建词表；载入现有词表\n", - "### 根据数据集构建词表" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [], - "source": [ - "# 初始化词表，该词表最大的vocab_size为10000，词表中每个词出现的最低频率为2，''表示未知词语，''表示padding词语\n", - "# Vocabulary默认初始化参数为max_size=None, min_freq=None, unknown='', padding=''\n", - "vocab = Vocabulary(max_size=10000, min_freq=2, unknown='', padding='')\n", - "\n", - "# 构建词表\n", - "train_data.apply(lambda x: [vocab.add(word) for word in x['premise']])\n", - "train_data.apply(lambda x: [vocab.add(word) for word in x['hypothesis']])\n", - "vocab.build_vocab()" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "({'premise': [2, 145, 146, 80, 147, 26, 148, 2, 104, 149, 150, 2, 151, 5, 55, 152, 105, 3] type=list,\n", - " 'hypothesis': [22, 80, 8, 1, 1, 20, 1, 3] type=list,\n", - " 'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'hypothesis_len': [1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'label': 2 type=int},\n", - " {'premise': [11, 5, 18, 5, 24, 6, 2, 10, 59, 52, 14, 9, 2, 53, 29, 60, 54, 45, 6, 46, 5, 7, 61, 3] type=list,\n", - " 'hypothesis': [22, 11, 1, 45, 3] type=list,\n", - " 'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'hypothesis_len': [1, 1, 1, 1, 1] type=list,\n", - " 'label': 1 type=int},\n", - " {'premise': [2, 11, 8, 14, 16, 7, 15, 50, 2, 66, 4, 76, 2, 10, 8, 98, 9, 58, 67, 3] type=list,\n", - " 'hypothesis': [22, 27, 50, 3] type=list,\n", - " 'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'hypothesis_len': [1, 1, 1, 1] type=list,\n", - " 'label': 0 type=int})" - ] - }, - "execution_count": 23, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 根据词表index句子\n", - "train_data.apply(lambda x: [vocab.to_index(word) for word in x['premise']], new_field_name='premise')\n", - "train_data.apply(lambda x: [vocab.to_index(word) for word in x['hypothesis']], new_field_name='hypothesis')\n", - "dev_data.apply(lambda x: [vocab.to_index(word) for word in x['premise']], new_field_name='premise')\n", - "dev_data.apply(lambda x: [vocab.to_index(word) for word in x['hypothesis']], new_field_name='hypothesis')\n", - "test_data.apply(lambda x: [vocab.to_index(word) for word in x['premise']], new_field_name='premise')\n", - "test_data.apply(lambda x: [vocab.to_index(word) for word in x['hypothesis']], new_field_name='hypothesis')\n", - "train_data[-1], dev_data[-1], test_data[-1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 载入现有词表\n", - "以BERT pretrained model为例，词表由一个vocab.txt文件来保存\n", - "用以下方法可以载入现有词表，并保证词表顺序不变" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [], - "source": [ - "# 读入vocab文件\n", - "with open('vocab.txt') as f:\n", - " lines = f.readlines()\n", - "vocabs = []\n", - "for line in lines:\n", - " vocabs.append(line.strip())\n", - "\n", - "# 实例化Vocabulary\n", - "vocab_bert = Vocabulary(unknown=None, padding=None)\n", - "# 将vocabs列表加入Vocabulary\n", - "vocab_bert.add_word_lst(vocabs)\n", - "# 构建词表\n", - "vocab_bert.build_vocab()\n", - "# 更新unknown与padding的token文本\n", - "vocab_bert.unknown = '[UNK]'\n", - "vocab_bert.padding = '[PAD]'" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "({'premise': [1037, 2210, 2223, 2136, 5363, 2000, 4608, 1037, 5479, 8058, 2046, 1037, 2918, 1999, 2019, 5027, 2208, 1012] type=list,\n", - " 'hypothesis': [100, 2136, 2003, 2652, 3598, 2006, 100, 1012] type=list,\n", - " 'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'hypothesis_len': [1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'label': 2 type=int},\n", - " {'premise': [2450, 1999, 2317, 1999, 100, 1998, 1037, 2158, 3621, 2369, 3788, 2007, 1037, 3696, 2005, 2198, 100, 10733, 1998, 100, 1999, 1996, 4281, 1012] type=list,\n", - " 'hypothesis': [100, 2450, 13063, 10733, 1012] type=list,\n", - " 'premise_len': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] type=list,\n", - " 'hypothesis_len': [1, 1, 1, 1, 1] type=list,\n", - " 'label': 1 type=int})" - ] - }, - "execution_count": 25, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 根据词表index句子\n", - "train_data_2.apply(lambda x: [vocab_bert.to_index(word) for word in x['premise']], new_field_name='premise')\n", - "train_data_2.apply(lambda x: [vocab_bert.to_index(word) for word in x['hypothesis']], new_field_name='hypothesis')\n", - "dev_data_2.apply(lambda x: [vocab_bert.to_index(word) for word in x['premise']], new_field_name='premise')\n", - "dev_data_2.apply(lambda x: [vocab_bert.to_index(word) for word in x['hypothesis']], new_field_name='hypothesis')\n", - "train_data_2[-1], dev_data_2[-1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 模型部分\n", - "## Model\n", - "模型部分fastNLP提供两种使用方式：调用fastNLP现有模型；开发者自行搭建模型\n", - "### 调用fastNLP现有模型" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'embed_dim': 300,\n", - " 'hidden_size': 300,\n", - " 'batch_first': True,\n", - " 'dropout': 0.3,\n", - " 'num_classes': 3,\n", - " 'gpu': True,\n", - " 'batch_size': 32,\n", - " 'vocab_size': 165}" - ] - }, - "execution_count": 26, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# step 1：加载模型参数（非必选）\n", - "from fastNLP.io.config_io import ConfigSection, ConfigLoader\n", - "args = ConfigSection()\n", - "ConfigLoader().load_config(\"./data/config\", {\"esim_model\": args})\n", - "args[\"vocab_size\"] = len(vocab)\n", - "args.data" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "ESIM(\n", - " (drop): Dropout(p=0.3)\n", - " (embedding): Embedding(\n", - " (embed): Embedding(165, 300, padding_idx=0)\n", - " (dropout): Dropout(p=0.3)\n", - " )\n", - " (embedding_layer): Linear(\n", - " (linear): Linear(in_features=300, out_features=300, bias=True)\n", - " )\n", - " (encoder): LSTM(\n", - " (lstm): LSTM(300, 300, batch_first=True, bidirectional=True)\n", - " )\n", - " (bi_attention): Bi_Attention()\n", - " (mean_pooling): MeanPoolWithMask()\n", - " (max_pooling): MaxPoolWithMask()\n", - " (inference_layer): Linear(\n", - " (linear): Linear(in_features=1200, out_features=300, bias=True)\n", - " )\n", - " (decoder): LSTM(\n", - " (lstm): LSTM(300, 300, batch_first=True, bidirectional=True)\n", - " )\n", - " (output): MLP(\n", - " (hiddens): ModuleList(\n", - " (0): Linear(in_features=1200, out_features=300, bias=True)\n", - " )\n", - " (output): Linear(in_features=300, out_features=3, bias=True)\n", - " (dropout): Dropout(p=0.3)\n", - " (hidden_active): Tanh()\n", - " )\n", - ")" - ] - }, - "execution_count": 27, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# step 2：加载ESIM模型\n", - "from fastNLP.models import ESIM\n", - "model = ESIM(**args.data)\n", - "model" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "CNNText(\n", - " (embed): Embedding(\n", - " (embed): Embedding(165, 50, padding_idx=0)\n", - " (dropout): Dropout(p=0.0)\n", - " )\n", - " (conv_pool): ConvMaxpool(\n", - " (convs): ModuleList(\n", - " (0): Conv1d(50, 3, kernel_size=(3,), stride=(1,), padding=(2,))\n", - " (1): Conv1d(50, 4, kernel_size=(4,), stride=(1,), padding=(2,))\n", - " (2): Conv1d(50, 5, kernel_size=(5,), stride=(1,), padding=(2,))\n", - " )\n", - " )\n", - " (dropout): Dropout(p=0.1)\n", - " (fc): Linear(\n", - " (linear): Linear(in_features=12, out_features=5, bias=True)\n", - " )\n", - ")" - ] - }, - "execution_count": 28, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# 另一个例子：加载CNN文本分类模型\n", - "from fastNLP.models import CNNText\n", - "cnn_text_model = CNNText(embed_num=len(vocab), embed_dim=50, num_classes=5, padding=2, dropout=0.1)\n", - "cnn_text_model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "这是上述模型的forward方法。如果你不知道什么是forward方法，请参考我们的PyTorch教程。\n", - "\n", - "注意两点：\n", - "1. forward参数名字叫**word_seq**，请记住。\n", - "2. forward的返回值是一个**dict**，其中有个key的名字叫**pred**。\n", - "\n", - "```Python\n", - " def forward(self, word_seq):\n", - " \"\"\"\n", - "\n", - " :param word_seq: torch.LongTensor, [batch_size, seq_len]\n", - " :return output: dict of torch.LongTensor, [batch_size, num_classes]\n", - " \"\"\"\n", - " x = self.embed(word_seq) # [N,L] -> [N,L,C]\n", - " x = self.conv_pool(x) # [N,L,C] -> [N,C]\n", - " x = self.dropout(x)\n", - " x = self.fc(x) # [N,C] -> [N, N_class]\n", - " return {'pred': x}\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 自行搭载模型\n", - "自行搭载的模型必须是nn.Module的子类， \n", - "(1)必须实现forward方法，并且forward方法不能出现**\\*arg**这种参数，例如\n", - "```Python\n", - " def forword(self, word_seq, *args): # 这是不允许的\n", - " xxx\n", - "```\n", - "forward函数的返回值必须是一个**dict**。 \n", - "dict当中模型预测的值所对应的key建议用**'pred'**，这里不做强制限制，但是如果不是pred的话，在加载loss函数的时候需要手动添加名称转换map \n", - "(2)如果实现了predict方法，在做evaluation的时候将调用predict方法而不是forward。如果没有predict方法，则在evaluation时调用forward方法。predict方法也不能使用\\*args这种参数形式，同时结果也必须返回一个dict，同样推荐key为'pred'。 \n", - "(3)forward函数可以计算loss并返回结果，在dict中的key为'loss'，如： \n", - "```Python\n", - " def forword(self, word_seq, *args): \n", - " xxx\n", - " return {'pred': pred, 'loss': loss}\n", - "```\n", - "当loss函数没有在trainer里被定义的时候，trainer将会根据forward函数返回的dict中key为'loss'的值来进行反向传播，具体见loss部分" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 训练测试部分\n", - "## Loss\n", - "### 键映射\n", - "根据上文DataSet与Model部分可以知道，fastNLP并不限制Model.forward()的返回值，也不限制DataSet中target field的key。因此在计算loss的时候，需要通过键映射的方式来完成取值。 \n", - "这里以CrossEntropyLoss为例，我们的交叉熵函数部分如下：\n", - "```Python\n", - " def get_loss(self, pred, target):\n", - " return F.cross_entropy(input=pred, target=target,\n", - " ignore_index=self.padding_idx)\n", - "```\n", - "这里接收的两个参数名字分别为pred和target，其中pred是从model的forward函数返回值中取得，target是从DataSet的is_target的field当中取得。在没有设置键映射的基础上，pred从model的forward函数返回的dict中取'pred'键得到；target从DataSet的'target'field中得到。\n", - "。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 修改键映射\n", - "在初始化CrossEntropyLoss的时候，可以传入两个参数(pred=None, target=None), 这两个参数接受的类型是str，假设(pred='output', target='label')，那么CrossEntropyLoss会使用'output'这个key在forward的output与batch_y中寻找值;'label'也是在forward的output与d ataset的is target field中寻找值。注意这里pred或target的来源并不一定非要来自于model.forward与dataset的is target field，也可以只来自于forward的结果" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 创建一个自己的loss\n", - "    (1)采用fastNLP.LossInForward，在model.forward()的结果中包含一个'loss'的key，具体见**自行搭载模型**部分 \n", - "    (2)自己定义一个继承于fastNLP.core.loss.LossBase的class。重写get_loss方法。 \n", - "      (2.1)在初始化自己的loss class的时候，需要初始化需要映射的值 \n", - "      (2.2)在get_loss函数中，参数的名字需要与初始化时的映射是一致的 \n", - "以L1Loss为例子：\n", - "```Python\n", - "class L1Loss(LossBase):\n", - " def __init__(self, pred=None, target=None):\n", - " super(L1Loss, self).__init__()\n", - " \"\"\"\n", - " 这里传入_init_param_map以使得pred和target被正确注册，但这一步不是必须的, 建议调用。传入_init_param_map的是用于\n", - " \"键映射\"的键值对。假设初始化__init__(pred=None, target=None, threshold=0.1)中threshold是用于控制loss计算的，\n", - " 则\\不要将threshold传入_init_param_map.\n", - " \"\"\"\n", - " self._init_param_map(pred=pred, target=target)\n", - "\n", - " def get_loss(self, pred, target):\n", - " # 这里'pred', 'target'必须和初始化的映射是一致的。\n", - " return F.l1_loss(input=pred, target=target)\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Trainer\n", - "trainer的作用是训练模型，是一个对训练过程的封装。trainer当中比较重要的函数是trainer.train()，train函数的主要步骤包括： \n", - "(1)创建batch \n", - "```Python\n", - "batch = Batch(dataset, batch_size, sampler=sampler)\n", - "```\n", - "(2)for batch_x, batch_y in batch: (batch_x, batch_y的内容分别为dataset中is input和is target的部分，这两个dict的key就是DataSet中的key，value会根据情况做好padding及tensor) \n", - "  (2.1)将batch_x, batch_y中的tensor移动到model所在的device \n", - "  (2.2)根据model.forward的参数列表，从batch_x中取出需要传递给forward的数据 \n", - "  (2.3)获取model.forward返回的dict，并与batch_y一起传递给loss函数，求得loss \n", - "  (2.4)对loss进行反向梯度传播并更新参数 \n", - "(3)如果有验证集，则进行验证 \n", - "(4)如果验证集的结果是当前最佳结果，则保存模型" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "除了以上的内容， Trainer中还提供了\"预跑\"的功能。该功能通过check_code_level管理，如果check_code_level为-1，则不进行\"预跑\"。 check_code_level=0，1，2代表不同的提醒级别。目前不同提醒级别对应的是对DataSet中设置为input或target但又没有使用的field的提醒级别。 0是忽略(默认)；1是会warning发生了未使用field的情况；2是出现了unused会直接报错并退出运行 \"预跑\"的主要目的有两个: \n", - "(1)防止train完了之后进行evaluation的时候出现错误。之前的train就白费了 \n", - "(2)由于存在\"键映射\"，直接运行导致的报错可能不太容易debug，通过\"预跑\"过程的报错会有一些debug提示 \"预跑\"会进行以下的操作： \n", - "  (i) 使用很小的batch_size, 检查batch_x中是否包含Model.forward所需要的参数。只会运行两个循环。 \n", - "  (ii)将Model.foward的输出pred_dict与batch_y输入到loss中，并尝试backward。不会更新参数，而且grad会被清零。如果传入了dev_data，还将进行metric的测试 \n", - "  (iii)创建Tester，并传入少量数据，检测是否可以正常运行 \n", - "\"预跑\"操作是在Trainer初始化的时候执行的。正常情况下，应该不需要改动\"预跑\"的代码。但如果遇到bug或者有什么好的建议，欢迎在开发群或者github提交issue。" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "training epochs started 2019-01-09 00-08-17\n", - "[tester] \n", - "AccuracyMetric: acc=0.206897\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/remote-home/ygxu/anaconda3/envs/no-fastnlp/lib/python3.7/site-packages/torch/nn/functional.py:1320: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n", - " warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[tester] \n", - "AccuracyMetric: acc=0.206897\n", - "[tester] \n", - "AccuracyMetric: acc=0.206897\n", - "[tester] \n", - "AccuracyMetric: acc=0.206897\n", - "[tester] \n", - "AccuracyMetric: acc=0.206897\n", - "\n", - "In Epoch:1/Step:4, got best dev performance:AccuracyMetric: acc=0.206897\n", - "Reloaded the best model.\n" - ] - }, - { - "data": { - "text/plain": [ - "{'best_eval': {'AccuracyMetric': {'acc': 0.206897}},\n", - " 'best_epoch': 1,\n", - " 'best_step': 4,\n", - " 'seconds': 0.79}" - ] - }, - "execution_count": 29, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from fastNLP import CrossEntropyLoss\n", - "from fastNLP import Adam\n", - "from fastNLP import AccuracyMetric\n", - "trainer = Trainer(\n", - " train_data=train_data,\n", - " model=model,\n", - " loss=CrossEntropyLoss(pred='pred', target='label'),\n", - " metrics=AccuracyMetric(),\n", - " n_epochs=5,\n", - " batch_size=16,\n", - " print_every=-1,\n", - " validate_every=-1,\n", - " dev_data=dev_data,\n", - " use_cuda=True,\n", - " optimizer=Adam(lr=1e-3, weight_decay=0),\n", - " check_code_level=-1,\n", - " metric_key='acc',\n", - " use_tqdm=False,\n", - ")\n", - "trainer.train()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Tester\n", - "tester的作用是训练模型，是一个对训练过程的封装。tester当中比较重要的函数是tester.test()，test函数的主要步骤包括：\n", - "(1)创建batch\n", - "```Python\n", - "batch = Batch(dataset, batch_size, sampler=sampler)\n", - "```\n", - "(2)for batch_x, batch_y in batch: (batch_x, batch_y的内容分别为dataset中is input和is target的部分，这两个dict的key就是DataSet中的key，value会根据情况做好padding及tensor) \n", - "  (2.1)同步数据与model将batch_x, batch_y中的tensor移动到model所在的device \n", - "  (2.2)根据predict_func的参数列表，从batch_x中取出需要传递给predict_func的数据，得到结果pred_dict \n", - "  (2.3)调用metric(pred_dict, batch_y) \n", - "  (2.4)当所有batch都运行完毕，会调用metric的get_metric方法，并且以返回的值作为evaluation的结果 " - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[tester] \n", - "AccuracyMetric: acc=0.263158\n" - ] - }, - { - "data": { - "text/plain": [ - "{'AccuracyMetric': {'acc': 0.263158}}" - ] - }, - "execution_count": 30, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "tester = Tester(\n", - " data=test_data,\n", - " model=model,\n", - " metrics=AccuracyMetric(),\n", - " batch_size=args[\"batch_size\"],\n", - ")\n", - "tester.test()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.7" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/tutorials/fastnlp_advanced_tutorial/data/config b/tutorials/fastnlp_advanced_tutorial/data/config deleted file mode 100644 index 87348b72..00000000 --- a/tutorials/fastnlp_advanced_tutorial/data/config +++ /dev/null @@ -1,8 +0,0 @@ -[esim_model] -embed_dim = 300 -hidden_size = 300 -batch_first = true -dropout = 0.3 -num_classes = 3 -gpu = true -batch_size = 32 diff --git a/tutorials/fastnlp_advanced_tutorial/hypothesis b/tutorials/fastnlp_advanced_tutorial/hypothesis deleted file mode 100644 index 07aec3c9..00000000 --- a/tutorials/fastnlp_advanced_tutorial/hypothesis +++ /dev/null @@ -1,100 +0,0 @@ -A person is training his horse for a competition . -A person is at a diner , ordering an omelette . -A person is outdoors , on a horse . -They are smiling at their parents -There are children present -The kids are frowning -The boy skates down the sidewalk . -The boy does a skateboarding trick . -The boy is wearing safety equipment . -An older man drinks his juice as he waits for his daughter to get off work . -A boy flips a burger . -An elderly man sits in a small shop . -Some women are hugging on vacation . -The women are sleeping . -There are women showing affection . -The people are eating omelettes . -The people are sitting at desks in school . -The diners are at a restaurant . -A man is drinking juice . -Two women are at a restaurant drinking wine . -A man in a restaurant is waiting for his meal to arrive . -A blond man getting a drink of water from a fountain in the park . -A blond man wearing a brown shirt is reading a book on a bench in the park -A blond man drinking water from a fountain . -The friends scowl at each other over a full dinner table . -There are two woman in this picture . -The friends have just met for the first time in 20 years , and have had a great time catching up . -The two sisters saw each other across the crowded diner and shared a hug , both clutching their doggie bags . -Two groups of rival gang members flipped each other off . -Two women hug each other . -A team is trying to score the games winning out . -A team is trying to tag a runner out . -A team is playing baseball on Saturn . -A school hosts a basketball game . -A high school is hosting an event . -A school is hosting an event . -The women do not care what clothes they wear . -Women are waiting by a tram . -The women enjoy having a good fashion sense . -A child with mom and dad , on summer vacation at the beach . -A family of three is at the beach . -A family of three is at the mall shopping . -The people waiting on the train are sitting . -There are people just getting on a train -There are people waiting on a train . -A couple are playing with a young child outside . -A couple are playing frisbee with a young child at the beach . -A couple watch a little girl play by herself on the beach . -The family is sitting down for dinner . -The family is outside . -The family is on vacation . -The people are standing still on the curb . -Near a couple of restaurants , two people walk across the street . -The couple are walking across the street together . -The woman is nake . -The woman is cold . -The woman is wearing green . -The man with the sign is caucasian . -They are protesting outside the capital . -A woman in white . -A man is advertising for a restaurant . -The woman is wearing black . -A man and a woman walk down a crowded city street . -The woman is wearing white . -They are working for John 's Pizza . -Olympic swimming . -A man and a soman are eating together at John 's Pizza and Gyro . -They are walking with a sign . -The woman is waiting for a friend . -The man is sitting down while he has a sign for John 's Pizza and Gyro in his arms . -The woman and man are outdoors . -A woman ordering pizza . -The people are related . -Two adults run across the street to get away from a red shirted person chasing them . -The adults are both male and female . -Two people walk home after a tasty steak dinner . -Two adults swimming in water -Two adults walk across a street . -Two people ride bicycles into a tunnel . -Two people walk away from a restaurant across a street . -Two adults walking across a road near the convicted prisoner dressed in red -Two friends cross a street . -Some people board a train . -Two adults walk across the street . -Two adults walking across a road -There are no women in the picture . -Two adults walk across the street to get away from a red shirted person who is chasing them . -A married couple is sleeping . -A female is next to a man . -A married couple is walking next to each other . -Nobody has food . -A woman eats a banana and walks across a street , and there is a man trailing behind her . -The woman and man are playing baseball together . -two coworkers cross pathes on a street -A woman eats ice cream walking down the sidewalk , and there is another woman in front of her with a purse . -The mans briefcase is for work . -A person eating . -A person that is hungry . -An actress and her favorite assistant talk a walk in the city . -a woman eating a banana crosses a street diff --git a/tutorials/fastnlp_advanced_tutorial/label b/tutorials/fastnlp_advanced_tutorial/label deleted file mode 100644 index e28836df..00000000 --- a/tutorials/fastnlp_advanced_tutorial/label +++ /dev/null @@ -1,100 +0,0 @@ -1 -2 -0 -1 -0 -2 -2 -0 -1 -1 -2 -1 -1 -2 -0 -1 -2 -0 -0 -2 -1 -1 -2 -0 -2 -0 -1 -1 -2 -0 -1 -0 -2 -2 -1 -0 -2 -0 -1 -1 -0 -2 -1 -0 -0 -0 -1 -2 -2 -0 -1 -2 -0 -1 -2 -1 -0 -1 -2 -0 -0 -2 -1 -0 -1 -2 -2 -0 -1 -2 -0 -1 -1 -2 -0 -1 -2 -0 -2 -0 -1 -1 -2 -0 -0 -2 -1 -2 -0 -1 -2 -0 -2 -1 -2 -1 -0 -1 -1 -0 diff --git a/tutorials/fastnlp_advanced_tutorial/premise b/tutorials/fastnlp_advanced_tutorial/premise deleted file mode 100644 index 0c9af30e..00000000 --- a/tutorials/fastnlp_advanced_tutorial/premise +++ /dev/null @@ -1,100 +0,0 @@ -A person on a horse jumps over a broken down airplane . -A person on a horse jumps over a broken down airplane . -A person on a horse jumps over a broken down airplane . -Children smiling and waving at camera -Children smiling and waving at camera -Children smiling and waving at camera -A boy is jumping on skateboard in the middle of a red bridge . -A boy is jumping on skateboard in the middle of a red bridge . -A boy is jumping on skateboard in the middle of a red bridge . -An older man sits with his orange juice at a small table in a coffee shop while employees in bright colored shirts smile in the background . -An older man sits with his orange juice at a small table in a coffee shop while employees in bright colored shirts smile in the background . -An older man sits with his orange juice at a small table in a coffee shop while employees in bright colored shirts smile in the background . -Two blond women are hugging one another . -Two blond women are hugging one another . -Two blond women are hugging one another . -A few people in a restaurant setting , one of them is drinking orange juice . -A few people in a restaurant setting , one of them is drinking orange juice . -A few people in a restaurant setting , one of them is drinking orange juice . -An older man is drinking orange juice at a restaurant . -An older man is drinking orange juice at a restaurant . -An older man is drinking orange juice at a restaurant . -A man with blond-hair , and a brown shirt drinking out of a public water fountain . -A man with blond-hair , and a brown shirt drinking out of a public water fountain . -A man with blond-hair , and a brown shirt drinking out of a public water fountain . -Two women who just had lunch hugging and saying goodbye . -Two women who just had lunch hugging and saying goodbye . -Two women who just had lunch hugging and saying goodbye . -Two women , holding food carryout containers , hug . -Two women , holding food carryout containers , hug . -Two women , holding food carryout containers , hug . -A Little League team tries to catch a runner sliding into a base in an afternoon game . -A Little League team tries to catch a runner sliding into a base in an afternoon game . -A Little League team tries to catch a runner sliding into a base in an afternoon game . -The school is having a special event in order to show the american culture on how other cultures are dealt with in parties . -The school is having a special event in order to show the american culture on how other cultures are dealt with in parties . -The school is having a special event in order to show the american culture on how other cultures are dealt with in parties . -High fashion ladies wait outside a tram beside a crowd of people in the city . -High fashion ladies wait outside a tram beside a crowd of people in the city . -High fashion ladies wait outside a tram beside a crowd of people in the city . -A man , woman , and child enjoying themselves on a beach . -A man , woman , and child enjoying themselves on a beach . -A man , woman , and child enjoying themselves on a beach . -People waiting to get on a train or just getting off . -People waiting to get on a train or just getting off . -People waiting to get on a train or just getting off . -A couple playing with a little boy on the beach . -A couple playing with a little boy on the beach . -A couple playing with a little boy on the beach . -A couple play in the tide with their young son . -A couple play in the tide with their young son . -A couple play in the tide with their young son . -A man and a woman cross the street in front of a pizza and gyro restaurant . -A man and a woman cross the street in front of a pizza and gyro restaurant . -A man and a woman cross the street in front of a pizza and gyro restaurant . -A woman in a green jacket and hood over her head looking towards a valley . -A woman in a green jacket and hood over her head looking towards a valley . -A woman in a green jacket and hood over her head looking towards a valley . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Woman in white in foreground and a man slightly behind walking with a sign for John 's Pizza and Gyro in the background . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -Two adults , one female in white , with shades and one male , gray clothes , walking across a street , away from a eatery with a blurred image of a dark colored red shirted person in the foreground . -A woman wearing all white and eating , walks next to a man holding a briefcase . -A woman wearing all white and eating , walks next to a man holding a briefcase . -A woman wearing all white and eating , walks next to a man holding a briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . -A woman is walking across the street eating a banana , while a man is following with his briefcase . diff --git a/tutorials/fastnlp_advanced_tutorial/tutorial_sample_dataset.csv b/tutorials/fastnlp_advanced_tutorial/tutorial_sample_dataset.csv deleted file mode 100644 index e5c0a74f..00000000 --- a/tutorials/fastnlp_advanced_tutorial/tutorial_sample_dataset.csv +++ /dev/null @@ -1,77 +0,0 @@ -A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . 1 -This quiet , introspective and entertaining independent is worth seeking . 4 -Even fans of Ismail Merchant 's work , I suspect , would have a hard time sitting through this one . 1 -A positively thrilling combination of ethnography and all the intrigue , betrayal , deceit and murder of a Shakespearean tragedy or a juicy soap opera . 3 -Aggressive self-glorification and a manipulative whitewash . 1 -A comedy-drama of nearly epic proportions rooted in a sincere performance by the title character undergoing midlife crisis . 4 -Narratively , Trouble Every Day is a plodding mess . 1 -The Importance of Being Earnest , so thick with wit it plays like a reading from Bartlett 's Familiar Quotations 3 -But it does n't leave you with much . 1 -You could hate it for the same reason . 1 -There 's little to recommend Snow Dogs , unless one considers cliched dialogue and perverse escapism a source of high hilarity . 1 -Kung Pow is Oedekerk 's realization of his childhood dream to be in a martial-arts flick , and proves that sometimes the dreams of youth should remain just that . 1 -The performances are an absolute joy . 4 -Fresnadillo has something serious to say about the ways in which extravagant chance can distort our perspective and throw us off the path of good sense . 3 -I still like Moonlight Mile , better judgment be damned . 3 -A welcome relief from baseball movies that try too hard to be mythic , this one is a sweet and modest and ultimately winning story . 3 -a bilingual charmer , just like the woman who inspired it 3 -Like a less dizzily gorgeous companion to Mr. Wong 's In the Mood for Love -- very much a Hong Kong movie despite its mainland setting . 2 -As inept as big-screen remakes of The Avengers and The Wild Wild West . 1 -It 's everything you 'd expect -- but nothing more . 2 -Best indie of the year , so far . 4 -Hatfield and Hicks make the oddest of couples , and in this sense the movie becomes a study of the gambles of the publishing world , offering a case study that exists apart from all the movie 's political ramifications . 3 -It 's like going to a house party and watching the host defend himself against a frothing ex-girlfriend . 1 -That the Chuck Norris `` grenade gag '' occurs about 7 times during Windtalkers is a good indication of how serious-minded the film is . 2 -The plot is romantic comedy boilerplate from start to finish . 2 -It arrives with an impeccable pedigree , mongrel pep , and almost indecipherable plot complications . 2 -A film that clearly means to preach exclusively to the converted . 2 -While The Importance of Being Earnest offers opportunities for occasional smiles and chuckles , it does n't give us a reason to be in the theater beyond Wilde 's wit and the actors ' performances . 1 -The latest vapid actor 's exercise to appropriate the structure of Arthur Schnitzler 's Reigen . 1 -More vaudeville show than well-constructed narrative , but on those terms it 's inoffensive and actually rather sweet . 2 -Nothing more than a run-of-the-mill action flick . 2 -Hampered -- no , paralyzed -- by a self-indulgent script ... that aims for poetry and ends up sounding like satire . 0 -Ice Age is the first computer-generated feature cartoon to feel like other movies , and that makes for some glacial pacing early on . 2 -There 's very little sense to what 's going on here , but the makers serve up the cliches with considerable dash . 2 -Cattaneo should have followed the runaway success of his first film , The Full Monty , with something different . 2 -They 're the unnamed , easily substitutable forces that serve as whatever terror the heroes of horror movies try to avoid . 1 -It almost feels as if the movie is more interested in entertaining itself than in amusing us . 1 -The movie 's progression into rambling incoherence gives new meaning to the phrase ` fatal script error . ' 0 -I still like Moonlight Mile , better judgment be damned . 3 -A welcome relief from baseball movies that try too hard to be mythic , this one is a sweet and modest and ultimately winning story . 3 -a bilingual charmer , just like the woman who inspired it 3 -Like a less dizzily gorgeous companion to Mr. Wong 's In the Mood for Love -- very much a Hong Kong movie despite its mainland setting . 2 -As inept as big-screen remakes of The Avengers and The Wild Wild West . 1 -It 's everything you 'd expect -- but nothing more . 2 -Best indie of the year , so far . 4 -Hatfield and Hicks make the oddest of couples , and in this sense the movie becomes a study of the gambles of the publishing world , offering a case study that exists apart from all the movie 's political ramifications . 3 -It 's like going to a house party and watching the host defend himself against a frothing ex-girlfriend . 1 -That the Chuck Norris `` grenade gag '' occurs about 7 times during Windtalkers is a good indication of how serious-minded the film is . 2 -The plot is romantic comedy boilerplate from start to finish . 2 -It arrives with an impeccable pedigree , mongrel pep , and almost indecipherable plot complications . 2 -A film that clearly means to preach exclusively to the converted . 2 -I still like Moonlight Mile , better judgment be damned . 3 -A welcome relief from baseball movies that try too hard to be mythic , this one is a sweet and modest and ultimately winning story . 3 -a bilingual charmer , just like the woman who inspired it 3 -Like a less dizzily gorgeous companion to Mr. Wong 's In the Mood for Love -- very much a Hong Kong movie despite its mainland setting . 2 -As inept as big-screen remakes of The Avengers and The Wild Wild West . 1 -It 's everything you 'd expect -- but nothing more . 2 -Best indie of the year , so far . 4 -Hatfield and Hicks make the oddest of couples , and in this sense the movie becomes a study of the gambles of the publishing world , offering a case study that exists apart from all the movie 's political ramifications . 3 -It 's like going to a house party and watching the host defend himself against a frothing ex-girlfriend . 1 -That the Chuck Norris `` grenade gag '' occurs about 7 times during Windtalkers is a good indication of how serious-minded the film is . 2 -The plot is romantic comedy boilerplate from start to finish . 2 -It arrives with an impeccable pedigree , mongrel pep , and almost indecipherable plot complications . 2 -A film that clearly means to preach exclusively to the converted . 2 -I still like Moonlight Mile , better judgment be damned . 3 -A welcome relief from baseball movies that try too hard to be mythic , this one is a sweet and modest and ultimately winning story . 3 -a bilingual charmer , just like the woman who inspired it 3 -Like a less dizzily gorgeous companion to Mr. Wong 's In the Mood for Love -- very much a Hong Kong movie despite its mainland setting . 2 -As inept as big-screen remakes of The Avengers and The Wild Wild West . 1 -It 's everything you 'd expect -- but nothing more . 2 -Best indie of the year , so far . 4 -Hatfield and Hicks make the oddest of couples , and in this sense the movie becomes a study of the gambles of the publishing world , offering a case study that exists apart from all the movie 's political ramifications . 3 -It 's like going to a house party and watching the host defend himself against a frothing ex-girlfriend . 1 -That the Chuck Norris `` grenade gag '' occurs about 7 times during Windtalkers is a good indication of how serious-minded the film is . 2 -The plot is romantic comedy boilerplate from start to finish . 2 -It arrives with an impeccable pedigree , mongrel pep , and almost indecipherable plot complications . 2 -A film that clearly means to preach exclusively to the converted . 2 \ No newline at end of file diff --git a/tutorials/fastnlp_advanced_tutorial/vocab.txt b/tutorials/fastnlp_advanced_tutorial/vocab.txt deleted file mode 100644 index fb140275..00000000 --- a/tutorials/fastnlp_advanced_tutorial/vocab.txt +++ /dev/null @@ -1,30522 +0,0 @@ -[PAD] -[unused0] -[unused1] -[unused2] -[unused3] -[unused4] -[unused5] -[unused6] -[unused7] -[unused8] -[unused9] -[unused10] -[unused11] -[unused12] -[unused13] -[unused14] -[unused15] -[unused16] -[unused17] -[unused18] -[unused19] -[unused20] -[unused21] -[unused22] -[unused23] -[unused24] -[unused25] -[unused26] -[unused27] -[unused28] -[unused29] -[unused30] -[unused31] -[unused32] -[unused33] -[unused34] -[unused35] -[unused36] -[unused37] -[unused38] -[unused39] -[unused40] -[unused41] -[unused42] -[unused43] -[unused44] -[unused45] -[unused46] -[unused47] -[unused48] -[unused49] -[unused50] -[unused51] -[unused52] -[unused53] -[unused54] -[unused55] -[unused56] -[unused57] -[unused58] -[unused59] -[unused60] -[unused61] -[unused62] -[unused63] -[unused64] -[unused65] -[unused66] -[unused67] -[unused68] -[unused69] -[unused70] -[unused71] -[unused72] -[unused73] -[unused74] -[unused75] -[unused76] -[unused77] -[unused78] -[unused79] -[unused80] -[unused81] -[unused82] -[unused83] -[unused84] -[unused85] -[unused86] -[unused87] -[unused88] -[unused89] -[unused90] -[unused91] -[unused92] -[unused93] -[unused94] -[unused95] -[unused96] -[unused97] -[unused98] -[UNK] -[CLS] -[SEP] -[MASK] -[unused99] -[unused100] -[unused101] -[unused102] -[unused103] -[unused104] -[unused105] -[unused106] -[unused107] -[unused108] -[unused109] -[unused110] -[unused111] -[unused112] -[unused113] -[unused114] -[unused115] -[unused116] -[unused117] -[unused118] -[unused119] -[unused120] -[unused121] -[unused122] -[unused123] -[unused124] -[unused125] -[unused126] -[unused127] -[unused128] -[unused129] -[unused130] -[unused131] -[unused132] -[unused133] -[unused134] -[unused135] -[unused136] -[unused137] -[unused138] -[unused139] -[unused140] -[unused141] -[unused142] -[unused143] -[unused144] -[unused145] -[unused146] -[unused147] -[unused148] -[unused149] -[unused150] -[unused151] -[unused152] -[unused153] -[unused154] -[unused155] -[unused156] -[unused157] -[unused158] -[unused159] -[unused160] -[unused161] -[unused162] -[unused163] -[unused164] -[unused165] -[unused166] -[unused167] -[unused168] -[unused169] -[unused170] -[unused171] -[unused172] -[unused173] -[unused174] -[unused175] -[unused176] -[unused177] -[unused178] -[unused179] -[unused180] -[unused181] -[unused182] -[unused183] -[unused184] -[unused185] -[unused186] -[unused187] -[unused188] -[unused189] -[unused190] -[unused191] -[unused192] -[unused193] -[unused194] -[unused195] -[unused196] -[unused197] -[unused198] -[unused199] -[unused200] -[unused201] -[unused202] -[unused203] -[unused204] -[unused205] -[unused206] -[unused207] -[unused208] -[unused209] -[unused210] -[unused211] -[unused212] -[unused213] -[unused214] -[unused215] -[unused216] -[unused217] -[unused218] -[unused219] -[unused220] -[unused221] -[unused222] -[unused223] -[unused224] -[unused225] -[unused226] -[unused227] -[unused228] -[unused229] -[unused230] -[unused231] -[unused232] -[unused233] -[unused234] -[unused235] -[unused236] -[unused237] -[unused238] -[unused239] -[unused240] -[unused241] -[unused242] -[unused243] -[unused244] -[unused245] -[unused246] -[unused247] -[unused248] -[unused249] -[unused250] -[unused251] -[unused252] -[unused253] -[unused254] -[unused255] -[unused256] -[unused257] -[unused258] -[unused259] -[unused260] -[unused261] -[unused262] -[unused263] -[unused264] -[unused265] -[unused266] -[unused267] -[unused268] -[unused269] -[unused270] -[unused271] -[unused272] -[unused273] -[unused274] -[unused275] -[unused276] -[unused277] -[unused278] -[unused279] -[unused280] -[unused281] -[unused282] -[unused283] -[unused284] -[unused285] -[unused286] -[unused287] -[unused288] -[unused289] -[unused290] -[unused291] -[unused292] -[unused293] -[unused294] -[unused295] -[unused296] -[unused297] -[unused298] -[unused299] -[unused300] -[unused301] -[unused302] -[unused303] -[unused304] -[unused305] -[unused306] -[unused307] -[unused308] -[unused309] -[unused310] -[unused311] -[unused312] -[unused313] -[unused314] -[unused315] -[unused316] -[unused317] -[unused318] -[unused319] -[unused320] -[unused321] -[unused322] -[unused323] -[unused324] -[unused325] -[unused326] -[unused327] -[unused328] -[unused329] -[unused330] -[unused331] -[unused332] -[unused333] -[unused334] -[unused335] -[unused336] -[unused337] -[unused338] -[unused339] -[unused340] -[unused341] -[unused342] -[unused343] -[unused344] -[unused345] -[unused346] -[unused347] -[unused348] -[unused349] -[unused350] -[unused351] -[unused352] -[unused353] -[unused354] -[unused355] -[unused356] -[unused357] -[unused358] -[unused359] -[unused360] -[unused361] -[unused362] -[unused363] -[unused364] -[unused365] -[unused366] -[unused367] -[unused368] -[unused369] -[unused370] -[unused371] -[unused372] -[unused373] -[unused374] -[unused375] -[unused376] -[unused377] -[unused378] -[unused379] -[unused380] -[unused381] -[unused382] -[unused383] -[unused384] -[unused385] -[unused386] -[unused387] -[unused388] -[unused389] -[unused390] -[unused391] -[unused392] -[unused393] -[unused394] -[unused395] -[unused396] -[unused397] -[unused398] -[unused399] -[unused400] -[unused401] -[unused402] -[unused403] -[unused404] -[unused405] -[unused406] -[unused407] -[unused408] -[unused409] -[unused410] -[unused411] -[unused412] -[unused413] -[unused414] -[unused415] -[unused416] -[unused417] -[unused418] -[unused419] -[unused420] -[unused421] -[unused422] -[unused423] -[unused424] -[unused425] -[unused426] -[unused427] -[unused428] -[unused429] -[unused430] -[unused431] -[unused432] -[unused433] -[unused434] -[unused435] -[unused436] -[unused437] -[unused438] -[unused439] -[unused440] -[unused441] -[unused442] -[unused443] -[unused444] -[unused445] -[unused446] -[unused447] -[unused448] -[unused449] -[unused450] -[unused451] -[unused452] -[unused453] -[unused454] -[unused455] -[unused456] -[unused457] -[unused458] -[unused459] -[unused460] -[unused461] -[unused462] -[unused463] -[unused464] -[unused465] -[unused466] -[unused467] -[unused468] -[unused469] -[unused470] -[unused471] -[unused472] -[unused473] -[unused474] -[unused475] -[unused476] -[unused477] -[unused478] -[unused479] -[unused480] -[unused481] -[unused482] -[unused483] -[unused484] -[unused485] -[unused486] -[unused487] -[unused488] -[unused489] -[unused490] -[unused491] -[unused492] -[unused493] -[unused494] -[unused495] -[unused496] -[unused497] -[unused498] -[unused499] -[unused500] -[unused501] -[unused502] -[unused503] -[unused504] -[unused505] -[unused506] -[unused507] -[unused508] -[unused509] -[unused510] -[unused511] -[unused512] -[unused513] -[unused514] -[unused515] -[unused516] -[unused517] -[unused518] -[unused519] -[unused520] -[unused521] -[unused522] -[unused523] -[unused524] -[unused525] -[unused526] -[unused527] -[unused528] -[unused529] -[unused530] -[unused531] -[unused532] -[unused533] -[unused534] -[unused535] -[unused536] -[unused537] -[unused538] -[unused539] -[unused540] -[unused541] -[unused542] -[unused543] -[unused544] -[unused545] -[unused546] -[unused547] -[unused548] -[unused549] -[unused550] -[unused551] -[unused552] -[unused553] -[unused554] -[unused555] -[unused556] -[unused557] -[unused558] -[unused559] -[unused560] -[unused561] -[unused562] -[unused563] -[unused564] -[unused565] -[unused566] -[unused567] -[unused568] -[unused569] -[unused570] -[unused571] -[unused572] -[unused573] -[unused574] -[unused575] -[unused576] -[unused577] -[unused578] -[unused579] -[unused580] -[unused581] -[unused582] -[unused583] -[unused584] -[unused585] -[unused586] -[unused587] -[unused588] -[unused589] -[unused590] -[unused591] -[unused592] -[unused593] -[unused594] -[unused595] -[unused596] -[unused597] -[unused598] -[unused599] -[unused600] -[unused601] -[unused602] -[unused603] -[unused604] -[unused605] -[unused606] -[unused607] -[unused608] -[unused609] -[unused610] -[unused611] -[unused612] -[unused613] -[unused614] -[unused615] -[unused616] -[unused617] -[unused618] -[unused619] -[unused620] -[unused621] -[unused622] -[unused623] -[unused624] -[unused625] -[unused626] -[unused627] -[unused628] -[unused629] -[unused630] -[unused631] -[unused632] -[unused633] -[unused634] -[unused635] -[unused636] -[unused637] -[unused638] -[unused639] -[unused640] -[unused641] -[unused642] -[unused643] -[unused644] -[unused645] -[unused646] -[unused647] -[unused648] -[unused649] -[unused650] -[unused651] -[unused652] -[unused653] -[unused654] -[unused655] -[unused656] -[unused657] -[unused658] -[unused659] -[unused660] -[unused661] -[unused662] -[unused663] -[unused664] -[unused665] -[unused666] -[unused667] -[unused668] -[unused669] -[unused670] -[unused671] -[unused672] -[unused673] -[unused674] -[unused675] -[unused676] -[unused677] -[unused678] -[unused679] -[unused680] -[unused681] -[unused682] -[unused683] -[unused684] -[unused685] -[unused686] -[unused687] -[unused688] -[unused689] -[unused690] -[unused691] -[unused692] -[unused693] -[unused694] -[unused695] -[unused696] -[unused697] -[unused698] -[unused699] -[unused700] -[unused701] -[unused702] -[unused703] -[unused704] -[unused705] -[unused706] -[unused707] -[unused708] -[unused709] -[unused710] -[unused711] -[unused712] -[unused713] -[unused714] -[unused715] -[unused716] -[unused717] -[unused718] -[unused719] -[unused720] -[unused721] -[unused722] -[unused723] -[unused724] -[unused725] -[unused726] -[unused727] -[unused728] -[unused729] -[unused730] -[unused731] -[unused732] -[unused733] -[unused734] -[unused735] -[unused736] -[unused737] -[unused738] -[unused739] -[unused740] -[unused741] -[unused742] -[unused743] -[unused744] -[unused745] -[unused746] -[unused747] -[unused748] -[unused749] -[unused750] -[unused751] -[unused752] -[unused753] -[unused754] -[unused755] -[unused756] -[unused757] -[unused758] -[unused759] -[unused760] -[unused761] -[unused762] -[unused763] -[unused764] -[unused765] -[unused766] -[unused767] -[unused768] -[unused769] -[unused770] -[unused771] -[unused772] -[unused773] -[unused774] -[unused775] -[unused776] -[unused777] -[unused778] -[unused779] -[unused780] -[unused781] -[unused782] -[unused783] -[unused784] -[unused785] -[unused786] -[unused787] -[unused788] -[unused789] -[unused790] -[unused791] -[unused792] -[unused793] -[unused794] -[unused795] -[unused796] -[unused797] -[unused798] -[unused799] -[unused800] -[unused801] -[unused802] -[unused803] -[unused804] -[unused805] -[unused806] -[unused807] -[unused808] -[unused809] -[unused810] -[unused811] -[unused812] -[unused813] -[unused814] -[unused815] -[unused816] -[unused817] -[unused818] -[unused819] -[unused820] -[unused821] -[unused822] -[unused823] -[unused824] -[unused825] -[unused826] -[unused827] -[unused828] -[unused829] -[unused830] -[unused831] -[unused832] -[unused833] -[unused834] -[unused835] -[unused836] -[unused837] -[unused838] -[unused839] -[unused840] -[unused841] -[unused842] -[unused843] -[unused844] -[unused845] -[unused846] -[unused847] -[unused848] -[unused849] -[unused850] -[unused851] -[unused852] -[unused853] -[unused854] -[unused855] -[unused856] -[unused857] -[unused858] -[unused859] -[unused860] -[unused861] -[unused862] -[unused863] -[unused864] -[unused865] -[unused866] -[unused867] -[unused868] -[unused869] -[unused870] -[unused871] -[unused872] -[unused873] -[unused874] -[unused875] -[unused876] -[unused877] -[unused878] -[unused879] -[unused880] -[unused881] -[unused882] -[unused883] -[unused884] -[unused885] -[unused886] -[unused887] -[unused888] -[unused889] -[unused890] -[unused891] -[unused892] -[unused893] -[unused894] -[unused895] -[unused896] -[unused897] -[unused898] -[unused899] -[unused900] -[unused901] -[unused902] -[unused903] -[unused904] -[unused905] -[unused906] -[unused907] -[unused908] -[unused909] -[unused910] -[unused911] -[unused912] -[unused913] -[unused914] -[unused915] -[unused916] -[unused917] -[unused918] -[unused919] -[unused920] -[unused921] -[unused922] -[unused923] -[unused924] -[unused925] -[unused926] -[unused927] -[unused928] -[unused929] -[unused930] -[unused931] -[unused932] -[unused933] -[unused934] -[unused935] -[unused936] -[unused937] -[unused938] -[unused939] -[unused940] -[unused941] -[unused942] -[unused943] -[unused944] -[unused945] -[unused946] -[unused947] -[unused948] -[unused949] -[unused950] -[unused951] -[unused952] -[unused953] -[unused954] -[unused955] -[unused956] -[unused957] -[unused958] -[unused959] -[unused960] -[unused961] -[unused962] -[unused963] -[unused964] -[unused965] -[unused966] -[unused967] -[unused968] -[unused969] -[unused970] -[unused971] -[unused972] -[unused973] -[unused974] -[unused975] -[unused976] -[unused977] -[unused978] -[unused979] -[unused980] -[unused981] -[unused982] -[unused983] -[unused984] -[unused985] -[unused986] -[unused987] -[unused988] -[unused989] -[unused990] -[unused991] -[unused992] -[unused993] -! -" -# -$ -% -& -' -( -) -* -+ -, -- -. -/ -0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -: -; -< -= -> -? -@ -[ -\ -] -^ -_ -` -a -b -c -d -e -f -g -h -i -j -k -l -m -n -o -p -q -r -s -t -u -v -w -x -y -z -{ -| -} -~ -¡ -¢ -£ -¤ -¥ -¦ -§ -¨ -© -ª -« -¬ -® -° -± -² -³ -´ -µ -¶ -· -¹ -º -» -¼ -½ -¾ -¿ -× -ß -æ -ð -÷ -ø -þ -đ -ħ -ı -ł -ŋ -œ -ƒ -ɐ -ɑ -ɒ -ɔ -ɕ -ə -ɛ -ɡ -ɣ -ɨ -ɪ -ɫ -ɬ -ɯ -ɲ -ɴ -ɹ -ɾ -ʀ -ʁ -ʂ -ʃ -ʉ -ʊ -ʋ -ʌ -ʎ -ʐ -ʑ -ʒ -ʔ -ʰ -ʲ -ʳ -ʷ -ʸ -ʻ -ʼ -ʾ -ʿ -ˈ -ː -ˡ -ˢ -ˣ -ˤ -α -β -γ -δ -ε -ζ -η -θ -ι -κ -λ -μ -ν -ξ -ο -π -ρ -ς -σ -τ -υ -φ -χ -ψ -ω -а -б -в -г -д -е -ж -з -и -к -л -м -н -о -п -р -с -т -у -ф -х -ц -ч -ш -щ -ъ -ы -ь -э -ю -я -ђ -є -і -ј -љ -њ -ћ -ӏ -ա -բ -գ -դ -ե -թ -ի -լ -կ -հ -մ -յ -ն -ո -պ -ս -վ -տ -ր -ւ -ք -־ -א -ב -ג -ד -ה -ו -ז -ח -ט -י -ך -כ -ל -ם -מ -ן -נ -ס -ע -ף -פ -ץ -צ -ק -ר -ש -ת -، -ء -ا -ب -ة -ت -ث -ج -ح -خ -د -ذ -ر -ز -س -ش -ص -ض -ط -ظ -ع -غ -ـ -ف -ق -ك -ل -م -ن -ه -و -ى -ي -ٹ -پ -چ -ک -گ -ں -ھ -ہ -ی -ے -अ -आ -उ -ए -क -ख -ग -च -ज -ट -ड -ण -त -थ -द -ध -न -प -ब -भ -म -य -र -ल -व -श -ष -स -ह -ा -ि -ी -ो -। -॥ -ং -অ -আ -ই -উ -এ -ও -ক -খ -গ -চ -ছ -জ -ট -ড -ণ -ত -থ -দ -ধ -ন -প -ব -ভ -ম -য -র -ল -শ -ষ -স -হ -া -ি -ী -ে -க -ச -ட -த -ந -ன -ப -ம -ய -ர -ல -ள -வ -ா -ி -ு -ே -ை -ನ -ರ -ಾ -ක -ය -ර -ල -ව -ා -ก -ง -ต -ท -น -พ -ม -ย -ร -ล -ว -ส -อ -า -เ -་ -། -ག -ང -ད -ན -པ -བ -མ -འ -ར -ལ -ས -မ -ა -ბ -გ -დ -ე -ვ -თ -ი -კ -ლ -მ -ნ -ო -რ -ს -ტ -უ -ᄀ -ᄂ -ᄃ -ᄅ -ᄆ -ᄇ -ᄉ -ᄊ -ᄋ -ᄌ -ᄎ -ᄏ -ᄐ -ᄑ -ᄒ -ᅡ -ᅢ -ᅥ -ᅦ -ᅧ -ᅩ -ᅪ -ᅭ -ᅮ -ᅯ -ᅲ -ᅳ -ᅴ -ᅵ -ᆨ -ᆫ -ᆯ -ᆷ -ᆸ -ᆼ -ᴬ -ᴮ -ᴰ -ᴵ -ᴺ -ᵀ -ᵃ -ᵇ -ᵈ -ᵉ -ᵍ -ᵏ -ᵐ -ᵒ -ᵖ -ᵗ -ᵘ -ᵢ -ᵣ -ᵤ -ᵥ -ᶜ -ᶠ -‐ -‑ -‒ -– -— -― -‖ -‘ -’ -‚ -“ -” -„ -† -‡ -• -… -‰ -′ -″ -› -‿ -⁄ -⁰ -ⁱ -⁴ -⁵ -⁶ -⁷ -⁸ -⁹ -⁺ -⁻ -ⁿ -₀ -₁ -₂ -₃ -₄ -₅ -₆ -₇ -₈ -₉ -₊ -₍ -₎ -ₐ -ₑ -ₒ -ₓ -ₕ -ₖ -ₗ -ₘ -ₙ -ₚ -ₛ -ₜ -₤ -₩ -€ -₱ -₹ -ℓ -№ -ℝ -™ -⅓ -⅔ -← -↑ -→ -↓ -↔ -↦ -⇄ -⇌ -⇒ -∂ -∅ -∆ -∇ -∈ -− -∗ -∘ -√ -∞ -∧ -∨ -∩ -∪ -≈ -≡ -≤ -≥ -⊂ -⊆ -⊕ -⊗ -⋅ -─ -│ -■ -▪ -● -★ -☆ -☉ -♠ -♣ -♥ -♦ -♭ -♯ -⟨ -⟩ -ⱼ -⺩ -⺼ -⽥ -、 -。 -〈 -〉 -《 -》 -「 -」 -『 -』 -〜 -あ -い -う -え -お -か -き -く -け -こ -さ -し -す -せ -そ -た -ち -っ -つ -て -と -な -に -ぬ -ね -の -は -ひ -ふ -へ -ほ -ま -み -む -め -も -や -ゆ -よ -ら -り -る -れ -ろ -を -ん -ァ -ア -ィ -イ -ウ -ェ -エ -オ -カ -キ -ク -ケ -コ -サ -シ -ス -セ -タ -チ -ッ -ツ -テ -ト -ナ -ニ -ノ -ハ -ヒ -フ -ヘ -ホ -マ -ミ -ム -メ -モ -ャ -ュ -ョ -ラ -リ -ル -レ -ロ -ワ -ン -・ -ー -一 -三 -上 -下 -不 -世 -中 -主 -久 -之 -也 -事 -二 -五 -井 -京 -人 -亻 -仁 -介 -代 -仮 -伊 -会 -佐 -侍 -保 -信 -健 -元 -光 -八 -公 -内 -出 -分 -前 -劉 -力 -加 -勝 -北 -区 -十 -千 -南 -博 -原 -口 -古 -史 -司 -合 -吉 -同 -名 -和 -囗 -四 -国 -國 -土 -地 -坂 -城 -堂 -場 -士 -夏 -外 -大 -天 -太 -夫 -奈 -女 -子 -学 -宀 -宇 -安 -宗 -定 -宣 -宮 -家 -宿 -寺 -將 -小 -尚 -山 -岡 -島 -崎 -川 -州 -巿 -帝 -平 -年 -幸 -广 -弘 -張 -彳 -後 -御 -德 -心 -忄 -志 -忠 -愛 -成 -我 -戦 -戸 -手 -扌 -政 -文 -新 -方 -日 -明 -星 -春 -昭 -智 -曲 -書 -月 -有 -朝 -木 -本 -李 -村 -東 -松 -林 -森 -楊 -樹 -橋 -歌 -止 -正 -武 -比 -氏 -民 -水 -氵 -氷 -永 -江 -沢 -河 -治 -法 -海 -清 -漢 -瀬 -火 -版 -犬 -王 -生 -田 -男 -疒 -発 -白 -的 -皇 -目 -相 -省 -真 -石 -示 -社 -神 -福 -禾 -秀 -秋 -空 -立 -章 -竹 -糹 -美 -義 -耳 -良 -艹 -花 -英 -華 -葉 -藤 -行 -街 -西 -見 -訁 -語 -谷 -貝 -貴 -車 -軍 -辶 -道 -郎 -郡 -部 -都 -里 -野 -金 -鈴 -镇 -長 -門 -間 -阝 -阿 -陳 -陽 -雄 -青 -面 -風 -食 -香 -馬 -高 -龍 -龸 -fi -fl -！ -（ -） -， -－ -． -／ -： -？ -～ -the -of -and -in -to -was -he -is -as -for -on -with -that -it -his -by -at -from -her -##s -she -you -had -an -were -but -be -this -are -not -my -they -one -which -or -have -him -me -first -all -also -their -has -up -who -out -been -when -after -there -into -new -two -its -##a -time -would -no -what -about -said -we -over -then -other -so -more -##e -can -if -like -back -them -only -some -could -##i -where -just -##ing -during -before -##n -do -##o -made -school -through -than -now -years -most -world -may -between -down -well -three -##d -year -while -will -##ed -##r -##y -later -##t -city -under -around -did -such -being -used -state -people -part -know -against -your -many -second -university -both -national -##er -these -don -known -off -way -until -re -how -even -get -head -... -didn -##ly -team -american -because -de -##l -born -united -film -since -still -long -work -south -us -became -any -high -again -day -family -see -right -man -eyes -house -season -war -states -including -took -life -north -same -each -called -name -much -place -however -go -four -group -another -found -won -area -here -going -10 -away -series -left -home -music -best -make -hand -number -company -several -never -last -john -000 -very -album -take -end -good -too -following -released -game -played -little -began -district -##m -old -want -those -side -held -own -early -county -ll -league -use -west -##u -face -think -##es -2010 -government -##h -march -came -small -general -town -june -##on -line -based -something -##k -september -thought -looked -along -international -2011 -air -july -club -went -january -october -our -august -april -york -12 -few -2012 -2008 -east -show -member -college -2009 -father -public -##us -come -men -five -set -station -church -##c -next -former -november -room -party -located -december -2013 -age -got -2007 -##g -system -let -love -2006 -though -every -2014 -look -song -water -century -without -body -black -night -within -great -women -single -ve -building -large -population -river -named -band -white -started -##an -once -15 -20 -should -18 -2015 -service -top -built -british -open -death -king -moved -local -times -children -february -book -why -11 -door -need -president -order -final -road -wasn -although -due -major -died -village -third -knew -2016 -asked -turned -st -wanted -say -##p -together -received -main -son -served -different -##en -behind -himself -felt -members -power -football -law -voice -play -##in -near -park -history -30 -having -2005 -16 -##man -saw -mother -##al -army -point -front -help -english -street -art -late -hands -games -award -##ia -young -14 -put -published -country -division -across -told -13 -often -ever -french -london -center -six -red -2017 -led -days -include -light -25 -find -tell -among -species -really -according -central -half -2004 -form -original -gave -office -making -enough -lost -full -opened -must -included -live -given -german -player -run -business -woman -community -cup -might -million -land -2000 -court -development -17 -short -round -ii -km -seen -class -story -always -become -sure -research -almost -director -council -la -##2 -career -things -using -island -##z -couldn -car -##is -24 -close -force -##1 -better -free -support -control -field -students -2003 -education -married -##b -nothing -worked -others -record -big -inside -level -anything -continued -give -james -##3 -military -established -non -returned -feel -does -title -written -thing -feet -william -far -co -association -hard -already -2002 -##ra -championship -human -western -100 -##na -department -hall -role -various -production -21 -19 -heart -2001 -living -fire -version -##ers -##f -television -royal -##4 -produced -working -act -case -society -region -present -radio -period -looking -least -total -keep -england -wife -program -per -brother -mind -special -22 -##le -am -works -soon -##6 -political -george -services -taken -created -##7 -further -able -reached -david -union -joined -upon -done -important -social -information -either -##ic -##x -appeared -position -ground -lead -rock -dark -election -23 -board -france -hair -course -arms -site -police -girl -instead -real -sound -##v -words -moment -##te -someone -##8 -summer -project -announced -san -less -wrote -past -followed -##5 -blue -founded -al -finally -india -taking -records -america -##ne -1999 -design -considered -northern -god -stop -battle -toward -european -outside -described -track -today -playing -language -28 -call -26 -heard -professional -low -australia -miles -california -win -yet -green -##ie -trying -blood -##ton -southern -science -maybe -everything -match -square -27 -mouth -video -race -recorded -leave -above -##9 -daughter -points -space -1998 -museum -change -middle -common -##0 -move -tv -post -##ta -lake -seven -tried -elected -closed -ten -paul -minister -##th -months -start -chief -return -canada -person -sea -release -similar -modern -brought -rest -hit -formed -mr -##la -1997 -floor -event -doing -thomas -1996 -robert -care -killed -training -star -week -needed -turn -finished -railway -rather -news -health -sent -example -ran -term -michael -coming -currently -yes -forces -despite -gold -areas -50 -stage -fact -29 -dead -says -popular -2018 -originally -germany -probably -developed -result -pulled -friend -stood -money -running -mi -signed -word -songs -child -eventually -met -tour -average -teams -minutes -festival -current -deep -kind -1995 -decided -usually -eastern -seemed -##ness -episode -bed -added -table -indian -private -charles -route -available -idea -throughout -centre -addition -appointed -style -1994 -books -eight -construction -press -mean -wall -friends -remained -schools -study -##ch -##um -institute -oh -chinese -sometimes -events -possible -1992 -australian -type -brown -forward -talk -process -food -debut -seat -performance -committee -features -character -arts -herself -else -lot -strong -russian -range -hours -peter -arm -##da -morning -dr -sold -##ry -quickly -directed -1993 -guitar -china -##w -31 -list -##ma -performed -media -uk -players -smile -##rs -myself -40 -placed -coach -province -towards -wouldn -leading -whole -boy -official -designed -grand -census -##el -europe -attack -japanese -henry -1991 -##re -##os -cross -getting -alone -action -lower -network -wide -washington -japan -1990 -hospital -believe -changed -sister -##ar -hold -gone -sir -hadn -ship -##ka -studies -academy -shot -rights -below -base -bad -involved -kept -largest -##ist -bank -future -especially -beginning -mark -movement -section -female -magazine -plan -professor -lord -longer -##ian -sat -walked -hill -actually -civil -energy -model -families -size -thus -aircraft -completed -includes -data -captain -##or -fight -vocals -featured -richard -bridge -fourth -1989 -officer -stone -hear -##ism -means -medical -groups -management -self -lips -competition -entire -lived -technology -leaving -federal -tournament -bit -passed -hot -independent -awards -kingdom -mary -spent -fine -doesn -reported -##ling -jack -fall -raised -itself -stay -true -studio -1988 -sports -replaced -paris -systems -saint -leader -theatre -whose -market -capital -parents -spanish -canadian -earth -##ity -cut -degree -writing -bay -christian -awarded -natural -higher -bill -##as -coast -provided -previous -senior -ft -valley -organization -stopped -onto -countries -parts -conference -queen -security -interest -saying -allowed -master -earlier -phone -matter -smith -winning -try -happened -moving -campaign -los -##ley -breath -nearly -mid -1987 -certain -girls -date -italian -african -standing -fell -artist -##ted -shows -deal -mine -industry -1986 -##ng -everyone -republic -provide -collection -library -student -##ville -primary -owned -older -via -heavy -1st -makes -##able -attention -anyone -africa -##ri -stated -length -ended -fingers -command -staff -skin -foreign -opening -governor -okay -medal -kill -sun -cover -job -1985 -introduced -chest -hell -feeling -##ies -success -meet -reason -standard -meeting -novel -1984 -trade -source -buildings -##land -rose -guy -goal -##ur -chapter -native -husband -previously -unit -limited -entered -weeks -producer -operations -mountain -takes -covered -forced -related -roman -complete -successful -key -texas -cold -##ya -channel -1980 -traditional -films -dance -clear -approximately -500 -nine -van -prince -question -active -tracks -ireland -regional -silver -author -personal -sense -operation -##ine -economic -1983 -holding -twenty -isbn -additional -speed -hour -edition -regular -historic -places -whom -shook -movie -km² -secretary -prior -report -chicago -read -foundation -view -engine -scored -1982 -units -ask -airport -property -ready -immediately -lady -month -listed -contract -##de -manager -themselves -lines -##ki -navy -writer -meant -##ts -runs -##ro -practice -championships -singer -glass -commission -required -forest -starting -culture -generally -giving -access -attended -test -couple -stand -catholic -martin -caught -executive -##less -eye -##ey -thinking -chair -quite -shoulder -1979 -hope -decision -plays -defeated -municipality -whether -structure -offered -slowly -pain -ice -direction -##ion -paper -mission -1981 -mostly -200 -noted -individual -managed -nature -lives -plant -##ha -helped -except -studied -computer -figure -relationship -issue -significant -loss -die -smiled -gun -ago -highest -1972 -##am -male -bring -goals -mexico -problem -distance -commercial -completely -location -annual -famous -drive -1976 -neck -1978 -surface -caused -italy -understand -greek -highway -wrong -hotel -comes -appearance -joseph -double -issues -musical -companies -castle -income -review -assembly -bass -initially -parliament -artists -experience -1974 -particular -walk -foot -engineering -talking -window -dropped -##ter -miss -baby -boys -break -1975 -stars -edge -remember -policy -carried -train -stadium -bar -sex -angeles -evidence -##ge -becoming -assistant -soviet -1977 -upper -step -wing -1970 -youth -financial -reach -##ll -actor -numerous -##se -##st -nodded -arrived -##ation -minute -##nt -believed -sorry -complex -beautiful -victory -associated -temple -1968 -1973 -chance -perhaps -metal -##son -1945 -bishop -##et -lee -launched -particularly -tree -le -retired -subject -prize -contains -yeah -theory -empire -##ce -suddenly -waiting -trust -recording -##to -happy -terms -camp -champion -1971 -religious -pass -zealand -names -2nd -port -ancient -tom -corner -represented -watch -legal -anti -justice -cause -watched -brothers -45 -material -changes -simply -response -louis -fast -##ting -answer -60 -historical -1969 -stories -straight -create -feature -increased -rate -administration -virginia -el -activities -cultural -overall -winner -programs -basketball -legs -guard -beyond -cast -doctor -mm -flight -results -remains -cost -effect -winter -##ble -larger -islands -problems -chairman -grew -commander -isn -1967 -pay -failed -selected -hurt -fort -box -regiment -majority -journal -35 -edward -plans -##ke -##ni -shown -pretty -irish -characters -directly -scene -likely -operated -allow -spring -##j -junior -matches -looks -mike -houses -fellow -##tion -beach -marriage -##ham -##ive -rules -oil -65 -florida -expected -nearby -congress -sam -peace -recent -iii -wait -subsequently -cell -##do -variety -serving -agreed -please -poor -joe -pacific -attempt -wood -democratic -piece -prime -##ca -rural -mile -touch -appears -township -1964 -1966 -soldiers -##men -##ized -1965 -pennsylvania -closer -fighting -claimed -score -jones -physical -editor -##ous -filled -genus -specific -sitting -super -mom -##va -therefore -supported -status -fear -cases -store -meaning -wales -minor -spain -tower -focus -vice -frank -follow -parish -separate -golden -horse -fifth -remaining -branch -32 -presented -stared -##id -uses -secret -forms -##co -baseball -exactly -##ck -choice -note -discovered -travel -composed -truth -russia -ball -color -kiss -dad -wind -continue -ring -referred -numbers -digital -greater -##ns -metres -slightly -direct -increase -1960 -responsible -crew -rule -trees -troops -##no -broke -goes -individuals -hundred -weight -creek -sleep -memory -defense -provides -ordered -code -value -jewish -windows -1944 -safe -judge -whatever -corps -realized -growing -pre -##ga -cities -alexander -gaze -lies -spread -scott -letter -showed -situation -mayor -transport -watching -workers -extended -##li -expression -normal -##ment -chart -multiple -border -##ba -host -##ner -daily -mrs -walls -piano -##ko -heat -cannot -##ate -earned -products -drama -era -authority -seasons -join -grade -##io -sign -difficult -machine -1963 -territory -mainly -##wood -stations -squadron -1962 -stepped -iron -19th -##led -serve -appear -sky -speak -broken -charge -knowledge -kilometres -removed -ships -article -campus -simple -##ty -pushed -britain -##ve -leaves -recently -cd -soft -boston -latter -easy -acquired -poland -##sa -quality -officers -presence -planned -nations -mass -broadcast -jean -share -image -influence -wild -offer -emperor -electric -reading -headed -ability -promoted -yellow -ministry -1942 -throat -smaller -politician -##by -latin -spoke -cars -williams -males -lack -pop -80 -##ier -acting -seeing -consists -##ti -estate -1961 -pressure -johnson -newspaper -jr -chris -olympics -online -conditions -beat -elements -walking -vote -##field -needs -carolina -text -featuring -global -block -shirt -levels -francisco -purpose -females -et -dutch -duke -ahead -gas -twice -safety -serious -turning -highly -lieutenant -firm -maria -amount -mixed -daniel -proposed -perfect -agreement -affairs -3rd -seconds -contemporary -paid -1943 -prison -save -kitchen -label -administrative -intended -constructed -academic -nice -teacher -races -1956 -formerly -corporation -ben -nation -issued -shut -1958 -drums -housing -victoria -seems -opera -1959 -graduated -function -von -mentioned -picked -build -recognized -shortly -protection -picture -notable -exchange -elections -1980s -loved -percent -racing -fish -elizabeth -garden -volume -hockey -1941 -beside -settled -##ford -1940 -competed -replied -drew -1948 -actress -marine -scotland -steel -glanced -farm -steve -1957 -risk -tonight -positive -magic -singles -effects -gray -screen -dog -##ja -residents -bus -sides -none -secondary -literature -polish -destroyed -flying -founder -households -1939 -lay -reserve -usa -gallery -##ler -1946 -industrial -younger -approach -appearances -urban -ones -1950 -finish -avenue -powerful -fully -growth -page -honor -jersey -projects -advanced -revealed -basic -90 -infantry -pair -equipment -visit -33 -evening -search -grant -effort -solo -treatment -buried -republican -primarily -bottom -owner -1970s -israel -gives -jim -dream -bob -remain -spot -70 -notes -produce -champions -contact -ed -soul -accepted -ways -del -##ally -losing -split -price -capacity -basis -trial -questions -##ina -1955 -20th -guess -officially -memorial -naval -initial -##ization -whispered -median -engineer -##ful -sydney -##go -columbia -strength -300 -1952 -tears -senate -00 -card -asian -agent -1947 -software -44 -draw -warm -supposed -com -pro -##il -transferred -leaned -##at -candidate -escape -mountains -asia -potential -activity -entertainment -seem -traffic -jackson -murder -36 -slow -product -orchestra -haven -agency -bbc -taught -website -comedy -unable -storm -planning -albums -rugby -environment -scientific -grabbed -protect -##hi -boat -typically -1954 -1953 -damage -principal -divided -dedicated -mount -ohio -##berg -pick -fought -driver -##der -empty -shoulders -sort -thank -berlin -prominent -account -freedom -necessary -efforts -alex -headquarters -follows -alongside -des -simon -andrew -suggested -operating -learning -steps -1949 -sweet -technical -begin -easily -34 -teeth -speaking -settlement -scale -##sh -renamed -ray -max -enemy -semi -joint -compared -##rd -scottish -leadership -analysis -offers -georgia -pieces -captured -animal -deputy -guest -organized -##lin -tony -combined -method -challenge -1960s -huge -wants -battalion -sons -rise -crime -types -facilities -telling -path -1951 -platform -sit -1990s -##lo -tells -assigned -rich -pull -##ot -commonly -alive -##za -letters -concept -conducted -wearing -happen -bought -becomes -holy -gets -ocean -defeat -languages -purchased -coffee -occurred -titled -##q -declared -applied -sciences -concert -sounds -jazz -brain -##me -painting -fleet -tax -nick -##ius -michigan -count -animals -leaders -episodes -##line -content -##den -birth -##it -clubs -64 -palace -critical -refused -fair -leg -laughed -returning -surrounding -participated -formation -lifted -pointed -connected -rome -medicine -laid -taylor -santa -powers -adam -tall -shared -focused -knowing -yards -entrance -falls -##wa -calling -##ad -sources -chosen -beneath -resources -yard -##ite -nominated -silence -zone -defined -##que -gained -thirty -38 -bodies -moon -##ard -adopted -christmas -widely -register -apart -iran -premier -serves -du -unknown -parties -##les -generation -##ff -continues -quick -fields -brigade -quiet -teaching -clothes -impact -weapons -partner -flat -theater -supreme -1938 -37 -relations -##tor -plants -suffered -1936 -wilson -kids -begins -##age -1918 -seats -armed -internet -models -worth -laws -400 -communities -classes -background -knows -thanks -quarter -reaching -humans -carry -killing -format -kong -hong -setting -75 -architecture -disease -railroad -inc -possibly -wish -arthur -thoughts -harry -doors -density -##di -crowd -illinois -stomach -tone -unique -reports -anyway -##ir -liberal -der -vehicle -thick -dry -drug -faced -largely -facility -theme -holds -creation -strange -colonel -##mi -revolution -bell -politics -turns -silent -rail -relief -independence -combat -shape -write -determined -sales -learned -4th -finger -oxford -providing -1937 -heritage -fiction -situated -designated -allowing -distribution -hosted -##est -sight -interview -estimated -reduced -##ria -toronto -footballer -keeping -guys -damn -claim -motion -sport -sixth -stayed -##ze -en -rear -receive -handed -twelve -dress -audience -granted -brazil -##well -spirit -##ated -noticed -etc -olympic -representative -eric -tight -trouble -reviews -drink -vampire -missing -roles -ranked -newly -household -finals -wave -critics -##ee -phase -massachusetts -pilot -unlike -philadelphia -bright -guns -crown -organizations -roof -42 -respectively -clearly -tongue -marked -circle -fox -korea -bronze -brian -expanded -sexual -supply -yourself -inspired -labour -fc -##ah -reference -vision -draft -connection -brand -reasons -1935 -classic -driving -trip -jesus -cells -entry -1920 -neither -trail -claims -atlantic -orders -labor -nose -afraid -identified -intelligence -calls -cancer -attacked -passing -stephen -positions -imperial -grey -jason -39 -sunday -48 -swedish -avoid -extra -uncle -message -covers -allows -surprise -materials -fame -hunter -##ji -1930 -citizens -figures -davis -environmental -confirmed -shit -titles -di -performing -difference -acts -attacks -##ov -existing -votes -opportunity -nor -shop -entirely -trains -opposite -pakistan -##pa -develop -resulted -representatives -actions -reality -pressed -##ish -barely -wine -conversation -faculty -northwest -ends -documentary -nuclear -stock -grace -sets -eat -alternative -##ps -bag -resulting -creating -surprised -cemetery -1919 -drop -finding -sarah -cricket -streets -tradition -ride -1933 -exhibition -target -ear -explained -rain -composer -injury -apartment -municipal -educational -occupied -netherlands -clean -billion -constitution -learn -1914 -maximum -classical -francis -lose -opposition -jose -ontario -bear -core -hills -rolled -ending -drawn -permanent -fun -##tes -##lla -lewis -sites -chamber -ryan -##way -scoring -height -1934 -##house -lyrics -staring -55 -officials -1917 -snow -oldest -##tic -orange -##ger -qualified -interior -apparently -succeeded -thousand -dinner -lights -existence -fans -heavily -41 -greatest -conservative -send -bowl -plus -enter -catch -##un -economy -duty -1929 -speech -authorities -princess -performances -versions -shall -graduate -pictures -effective -remembered -poetry -desk -crossed -starring -starts -passenger -sharp -##ant -acres -ass -weather -falling -rank -fund -supporting -check -adult -publishing -heads -cm -southeast -lane -##burg -application -bc -##ura -les -condition -transfer -prevent -display -ex -regions -earl -federation -cool -relatively -answered -besides -1928 -obtained -portion -##town -mix -##ding -reaction -liked -dean -express -peak -1932 -##tte -counter -religion -chain -rare -miller -convention -aid -lie -vehicles -mobile -perform -squad -wonder -lying -crazy -sword -##ping -attempted -centuries -weren -philosophy -category -##ize -anna -interested -47 -sweden -wolf -frequently -abandoned -kg -literary -alliance -task -entitled -##ay -threw -promotion -factory -tiny -soccer -visited -matt -fm -achieved -52 -defence -internal -persian -43 -methods -##ging -arrested -otherwise -cambridge -programming -villages -elementary -districts -rooms -criminal -conflict -worry -trained -1931 -attempts -waited -signal -bird -truck -subsequent -programme -##ol -ad -49 -communist -details -faith -sector -patrick -carrying -laugh -##ss -controlled -korean -showing -origin -fuel -evil -1927 -##ent -brief -identity -darkness -address -pool -missed -publication -web -planet -ian -anne -wings -invited -##tt -briefly -standards -kissed -##be -ideas -climate -causing -walter -worse -albert -articles -winners -desire -aged -northeast -dangerous -gate -doubt -1922 -wooden -multi -##ky -poet -rising -funding -46 -communications -communication -violence -copies -prepared -ford -investigation -skills -1924 -pulling -electronic -##ak -##ial -##han -containing -ultimately -offices -singing -understanding -restaurant -tomorrow -fashion -christ -ward -da -pope -stands -5th -flow -studios -aired -commissioned -contained -exist -fresh -americans -##per -wrestling -approved -kid -employed -respect -suit -1925 -angel -asking -increasing -frame -angry -selling -1950s -thin -finds -##nd -temperature -statement -ali -explain -inhabitants -towns -extensive -narrow -51 -jane -flowers -images -promise -somewhere -object -fly -closely -##ls -1912 -bureau -cape -1926 -weekly -presidential -legislative -1921 -##ai -##au -launch -founding -##ny -978 -##ring -artillery -strike -un -institutions -roll -writers -landing -chose -kevin -anymore -pp -##ut -attorney -fit -dan -billboard -receiving -agricultural -breaking -sought -dave -admitted -lands -mexican -##bury -charlie -specifically -hole -iv -howard -credit -moscow -roads -accident -1923 -proved -wear -struck -hey -guards -stuff -slid -expansion -1915 -cat -anthony -##kin -melbourne -opposed -sub -southwest -architect -failure -plane -1916 -##ron -map -camera -tank -listen -regarding -wet -introduction -metropolitan -link -ep -fighter -inch -grown -gene -anger -fixed -buy -dvd -khan -domestic -worldwide -chapel -mill -functions -examples -##head -developing -1910 -turkey -hits -pocket -antonio -papers -grow -unless -circuit -18th -concerned -attached -journalist -selection -journey -converted -provincial -painted -hearing -aren -bands -negative -aside -wondered -knight -lap -survey -ma -##ow -noise -billy -##ium -shooting -guide -bedroom -priest -resistance -motor -homes -sounded -giant -##mer -150 -scenes -equal -comic -patients -hidden -solid -actual -bringing -afternoon -touched -funds -wedding -consisted -marie -canal -sr -kim -treaty -turkish -recognition -residence -cathedral -broad -knees -incident -shaped -fired -norwegian -handle -cheek -contest -represent -##pe -representing -beauty -##sen -birds -advantage -emergency -wrapped -drawing -notice -pink -broadcasting -##ong -somehow -bachelor -seventh -collected -registered -establishment -alan -assumed -chemical -personnel -roger -retirement -jeff -portuguese -wore -tied -device -threat -progress -advance -##ised -banks -hired -manchester -nfl -teachers -structures -forever -##bo -tennis -helping -saturday -sale -applications -junction -hip -incorporated -neighborhood -dressed -ceremony -##ds -influenced -hers -visual -stairs -decades -inner -kansas -hung -hoped -gain -scheduled -downtown -engaged -austria -clock -norway -certainly -pale -protected -1913 -victor -employees -plate -putting -surrounded -##ists -finishing -blues -tropical -##ries -minnesota -consider -philippines -accept -54 -retrieved -1900 -concern -anderson -properties -institution -gordon -successfully -vietnam -##dy -backing -outstanding -muslim -crossing -folk -producing -usual -demand -occurs -observed -lawyer -educated -##ana -kelly -string -pleasure -budget -items -quietly -colorado -philip -typical -##worth -derived -600 -survived -asks -mental -##ide -56 -jake -jews -distinguished -ltd -1911 -sri -extremely -53 -athletic -loud -thousands -worried -shadow -transportation -horses -weapon -arena -importance -users -tim -objects -contributed -dragon -douglas -aware -senator -johnny -jordan -sisters -engines -flag -investment -samuel -shock -capable -clark -row -wheel -refers -session -familiar -biggest -wins -hate -maintained -drove -hamilton -request -expressed -injured -underground -churches -walker -wars -tunnel -passes -stupid -agriculture -softly -cabinet -regarded -joining -indiana -##ea -##ms -push -dates -spend -behavior -woods -protein -gently -chase -morgan -mention -burning -wake -combination -occur -mirror -leads -jimmy -indeed -impossible -singapore -paintings -covering -##nes -soldier -locations -attendance -sell -historian -wisconsin -invasion -argued -painter -diego -changing -egypt -##don -experienced -inches -##ku -missouri -vol -grounds -spoken -switzerland -##gan -reform -rolling -ha -forget -massive -resigned -burned -allen -tennessee -locked -values -improved -##mo -wounded -universe -sick -dating -facing -pack -purchase -user -##pur -moments -##ul -merged -anniversary -1908 -coal -brick -understood -causes -dynasty -queensland -establish -stores -crisis -promote -hoping -views -cards -referee -extension -##si -raise -arizona -improve -colonial -formal -charged -##rt -palm -lucky -hide -rescue -faces -95 -feelings -candidates -juan -##ell -goods -6th -courses -weekend -59 -luke -cash -fallen -##om -delivered -affected -installed -carefully -tries -swiss -hollywood -costs -lincoln -responsibility -##he -shore -file -proper -normally -maryland -assistance -jump -constant -offering -friendly -waters -persons -realize -contain -trophy -800 -partnership -factor -58 -musicians -cry -bound -oregon -indicated -hero -houston -medium -##ure -consisting -somewhat -##ara -57 -cycle -##che -beer -moore -frederick -gotten -eleven -worst -weak -approached -arranged -chin -loan -universal -bond -fifteen -pattern -disappeared -##ney -translated -##zed -lip -arab -capture -interests -insurance -##chi -shifted -cave -prix -warning -sections -courts -coat -plot -smell -feed -golf -favorite -maintain -knife -vs -voted -degrees -finance -quebec -opinion -translation -manner -ruled -operate -productions -choose -musician -discovery -confused -tired -separated -stream -techniques -committed -attend -ranking -kings -throw -passengers -measure -horror -fan -mining -sand -danger -salt -calm -decade -dam -require -runner -##ik -rush -associate -greece -##ker -rivers -consecutive -matthew -##ski -sighed -sq -documents -steam -edited -closing -tie -accused -1905 -##ini -islamic -distributed -directors -organisation -bruce -7th -breathing -mad -lit -arrival -concrete -taste -08 -composition -shaking -faster -amateur -adjacent -stating -1906 -twin -flew -##ran -tokyo -publications -##tone -obviously -ridge -storage -1907 -carl -pages -concluded -desert -driven -universities -ages -terminal -sequence -borough -250 -constituency -creative -cousin -economics -dreams -margaret -notably -reduce -montreal -mode -17th -ears -saved -jan -vocal -##ica -1909 -andy -##jo -riding -roughly -threatened -##ise -meters -meanwhile -landed -compete -repeated -grass -czech -regularly -charges -tea -sudden -appeal -##ung -solution -describes -pierre -classification -glad -parking -##ning -belt -physics -99 -rachel -add -hungarian -participate -expedition -damaged -gift -childhood -85 -fifty -##red -mathematics -jumped -letting -defensive -mph -##ux -##gh -testing -##hip -hundreds -shoot -owners -matters -smoke -israeli -kentucky -dancing -mounted -grandfather -emma -designs -profit -argentina -##gs -truly -li -lawrence -cole -begun -detroit -willing -branches -smiling -decide -miami -enjoyed -recordings -##dale -poverty -ethnic -gay -##bi -gary -arabic -09 -accompanied -##one -##ons -fishing -determine -residential -acid -##ary -alice -returns -starred -mail -##ang -jonathan -strategy -##ue -net -forty -cook -businesses -equivalent -commonwealth -distinct -ill -##cy -seriously -##ors -##ped -shift -harris -replace -rio -imagine -formula -ensure -##ber -additionally -scheme -conservation -occasionally -purposes -feels -favor -##and -##ore -1930s -contrast -hanging -hunt -movies -1904 -instruments -victims -danish -christopher -busy -demon -sugar -earliest -colony -studying -balance -duties -##ks -belgium -slipped -carter -05 -visible -stages -iraq -fifa -##im -commune -forming -zero -07 -continuing -talked -counties -legend -bathroom -option -tail -clay -daughters -afterwards -severe -jaw -visitors -##ded -devices -aviation -russell -kate -##vi -entering -subjects -##ino -temporary -swimming -forth -smooth -ghost -audio -bush -operates -rocks -movements -signs -eddie -##tz -ann -voices -honorary -06 -memories -dallas -pure -measures -racial -promised -66 -harvard -ceo -16th -parliamentary -indicate -benefit -flesh -dublin -louisiana -1902 -1901 -patient -sleeping -1903 -membership -coastal -medieval -wanting -element -scholars -rice -62 -limit -survive -makeup -rating -definitely -collaboration -obvious -##tan -boss -ms -baron -birthday -linked -soil -diocese -##lan -ncaa -##mann -offensive -shell -shouldn -waist -##tus -plain -ross -organ -resolution -manufacturing -adding -relative -kennedy -98 -whilst -moth -marketing -gardens -crash -72 -heading -partners -credited -carlos -moves -cable -##zi -marshall -##out -depending -bottle -represents -rejected -responded -existed -04 -jobs -denmark -lock -##ating -treated -graham -routes -talent -commissioner -drugs -secure -tests -reign -restored -photography -##gi -contributions -oklahoma -designer -disc -grin -seattle -robin -paused -atlanta -unusual -##gate -praised -las -laughing -satellite -hungary -visiting -##sky -interesting -factors -deck -poems -norman -##water -stuck -speaker -rifle -domain -premiered -##her -dc -comics -actors -01 -reputation -eliminated -8th -ceiling -prisoners -script -##nce -leather -austin -mississippi -rapidly -admiral -parallel -charlotte -guilty -tools -gender -divisions -fruit -##bs -laboratory -nelson -fantasy -marry -rapid -aunt -tribe -requirements -aspects -suicide -amongst -adams -bone -ukraine -abc -kick -sees -edinburgh -clothing -column -rough -gods -hunting -broadway -gathered -concerns -##ek -spending -ty -12th -snapped -requires -solar -bones -cavalry -##tta -iowa -drinking -waste -index -franklin -charity -thompson -stewart -tip -flash -landscape -friday -enjoy -singh -poem -listening -##back -eighth -fred -differences -adapted -bomb -ukrainian -surgery -corporate -masters -anywhere -##more -waves -odd -sean -portugal -orleans -dick -debate -kent -eating -puerto -cleared -96 -expect -cinema -97 -guitarist -blocks -electrical -agree -involving -depth -dying -panel -struggle -##ged -peninsula -adults -novels -emerged -vienna -metro -debuted -shoes -tamil -songwriter -meets -prove -beating -instance -heaven -scared -sending -marks -artistic -passage -superior -03 -significantly -shopping -##tive -retained -##izing -malaysia -technique -cheeks -##ola -warren -maintenance -destroy -extreme -allied -120 -appearing -##yn -fill -advice -alabama -qualifying -policies -cleveland -hat -battery -smart -authors -10th -soundtrack -acted -dated -lb -glance -equipped -coalition -funny -outer -ambassador -roy -possibility -couples -campbell -dna -loose -ethan -supplies -1898 -gonna -88 -monster -##res -shake -agents -frequency -springs -dogs -practices -61 -gang -plastic -easier -suggests -gulf -blade -exposed -colors -industries -markets -pan -nervous -electoral -charts -legislation -ownership -##idae -mac -appointment -shield -copy -assault -socialist -abbey -monument -license -throne -employment -jay -93 -replacement -charter -cloud -powered -suffering -accounts -oak -connecticut -strongly -wright -colour -crystal -13th -context -welsh -networks -voiced -gabriel -jerry -##cing -forehead -mp -##ens -manage -schedule -totally -remix -##ii -forests -occupation -print -nicholas -brazilian -strategic -vampires -engineers -76 -roots -seek -correct -instrumental -und -alfred -backed -hop -##des -stanley -robinson -traveled -wayne -welcome -austrian -achieve -67 -exit -rates -1899 -strip -whereas -##cs -sing -deeply -adventure -bobby -rick -jamie -careful -components -cap -useful -personality -knee -##shi -pushing -hosts -02 -protest -ca -ottoman -symphony -##sis -63 -boundary -1890 -processes -considering -considerable -tons -##work -##ft -##nia -cooper -trading -dear -conduct -91 -illegal -apple -revolutionary -holiday -definition -harder -##van -jacob -circumstances -destruction -##lle -popularity -grip -classified -liverpool -donald -baltimore -flows -seeking -honour -approval -92 -mechanical -till -happening -statue -critic -increasingly -immediate -describe -commerce -stare -##ster -indonesia -meat -rounds -boats -baker -orthodox -depression -formally -worn -naked -claire -muttered -sentence -11th -emily -document -77 -criticism -wished -vessel -spiritual -bent -virgin -parker -minimum -murray -lunch -danny -printed -compilation -keyboards -false -blow -belonged -68 -raising -78 -cutting -##board -pittsburgh -##up -9th -shadows -81 -hated -indigenous -jon -15th -barry -scholar -ah -##zer -oliver -##gy -stick -susan -meetings -attracted -spell -romantic -##ver -ye -1895 -photo -demanded -customers -##ac -1896 -logan -revival -keys -modified -commanded -jeans -##ious -upset -raw -phil -detective -hiding -resident -vincent -##bly -experiences -diamond -defeating -coverage -lucas -external -parks -franchise -helen -bible -successor -percussion -celebrated -il -lift -profile -clan -romania -##ied -mills -##su -nobody -achievement -shrugged -fault -1897 -rhythm -initiative -breakfast -carbon -700 -69 -lasted -violent -74 -wound -ken -killer -gradually -filmed -°c -dollars -processing -94 -remove -criticized -guests -sang -chemistry -##vin -legislature -disney -##bridge -uniform -escaped -integrated -proposal -purple -denied -liquid -karl -influential -morris -nights -stones -intense -experimental -twisted -71 -84 -##ld -pace -nazi -mitchell -ny -blind -reporter -newspapers -14th -centers -burn -basin -forgotten -surviving -filed -collections -monastery -losses -manual -couch -description -appropriate -merely -tag -missions -sebastian -restoration -replacing -triple -73 -elder -julia -warriors -benjamin -julian -convinced -stronger -amazing -declined -versus -merchant -happens -output -finland -bare -barbara -absence -ignored -dawn -injuries -##port -producers -##ram -82 -luis -##ities -kw -admit -expensive -electricity -nba -exception -symbol -##ving -ladies -shower -sheriff -characteristics -##je -aimed -button -ratio -effectively -summit -angle -jury -bears -foster -vessels -pants -executed -evans -dozen -advertising -kicked -patrol -1889 -competitions -lifetime -principles -athletics -##logy -birmingham -sponsored -89 -rob -nomination -1893 -acoustic -##sm -creature -longest -##tra -credits -harbor -dust -josh -##so -territories -milk -infrastructure -completion -thailand -indians -leon -archbishop -##sy -assist -pitch -blake -arrangement -girlfriend -serbian -operational -hence -sad -scent -fur -dj -sessions -hp -refer -rarely -##ora -exists -1892 -##ten -scientists -dirty -penalty -burst -portrait -seed -79 -pole -limits -rival -1894 -stable -alpha -grave -constitutional -alcohol -arrest -flower -mystery -devil -architectural -relationships -greatly -habitat -##istic -larry -progressive -remote -cotton -##ics -##ok -preserved -reaches -##ming -cited -86 -vast -scholarship -decisions -cbs -joy -teach -1885 -editions -knocked -eve -searching -partly -participation -gap -animated -fate -excellent -##ett -na -87 -alternate -saints -youngest -##ily -climbed -##ita -##tors -suggest -##ct -discussion -staying -choir -lakes -jacket -revenue -nevertheless -peaked -instrument -wondering -annually -managing -neil -1891 -signing -terry -##ice -apply -clinical -brooklyn -aim -catherine -fuck -farmers -figured -ninth -pride -hugh -evolution -ordinary -involvement -comfortable -shouted -tech -encouraged -taiwan -representation -sharing -##lia -##em -panic -exact -cargo -competing -fat -cried -83 -1920s -occasions -pa -cabin -borders -utah -marcus -##isation -badly -muscles -##ance -victorian -transition -warner -bet -permission -##rin -slave -terrible -similarly -shares -seth -uefa -possession -medals -benefits -colleges -lowered -perfectly -mall -transit -##ye -##kar -publisher -##ened -harrison -deaths -elevation -##ae -asleep -machines -sigh -ash -hardly -argument -occasion -parent -leo -decline -1888 -contribution -##ua -concentration -1000 -opportunities -hispanic -guardian -extent -emotions -hips -mason -volumes -bloody -controversy -diameter -steady -mistake -phoenix -identify -violin -##sk -departure -richmond -spin -funeral -enemies -1864 -gear -literally -connor -random -sergeant -grab -confusion -1865 -transmission -informed -op -leaning -sacred -suspended -thinks -gates -portland -luck -agencies -yours -hull -expert -muscle -layer -practical -sculpture -jerusalem -latest -lloyd -statistics -deeper -recommended -warrior -arkansas -mess -supports -greg -eagle -1880 -recovered -rated -concerts -rushed -##ano -stops -eggs -files -premiere -keith -##vo -delhi -turner -pit -affair -belief -paint -##zing -mate -##ach -##ev -victim -##ology -withdrew -bonus -styles -fled -##ud -glasgow -technologies -funded -nbc -adaptation -##ata -portrayed -cooperation -supporters -judges -bernard -justin -hallway -ralph -##ick -graduating -controversial -distant -continental -spider -bite -##ho -recognize -intention -mixing -##ese -egyptian -bow -tourism -suppose -claiming -tiger -dominated -participants -vi -##ru -nurse -partially -tape -##rum -psychology -##rn -essential -touring -duo -voting -civilian -emotional -channels -##king -apparent -hebrew -1887 -tommy -carrier -intersection -beast -hudson -##gar -##zo -lab -nova -bench -discuss -costa -##ered -detailed -behalf -drivers -unfortunately -obtain -##lis -rocky -##dae -siege -friendship -honey -##rian -1861 -amy -hang -posted -governments -collins -respond -wildlife -preferred -operator -##po -laura -pregnant -videos -dennis -suspected -boots -instantly -weird -automatic -businessman -alleged -placing -throwing -ph -mood -1862 -perry -venue -jet -remainder -##lli -##ci -passion -biological -boyfriend -1863 -dirt -buffalo -ron -segment -fa -abuse -##era -genre -thrown -stroke -colored -stress -exercise -displayed -##gen -struggled -##tti -abroad -dramatic -wonderful -thereafter -madrid -component -widespread -##sed -tale -citizen -todd -monday -1886 -vancouver -overseas -forcing -crying -descent -##ris -discussed -substantial -ranks -regime -1870 -provinces -switch -drum -zane -ted -tribes -proof -lp -cream -researchers -volunteer -manor -silk -milan -donated -allies -venture -principle -delivery -enterprise -##ves -##ans -bars -traditionally -witch -reminded -copper -##uk -pete -inter -links -colin -grinned -elsewhere -competitive -frequent -##oy -scream -##hu -tension -texts -submarine -finnish -defending -defend -pat -detail -1884 -affiliated -stuart -themes -villa -periods -tool -belgian -ruling -crimes -answers -folded -licensed -resort -demolished -hans -lucy -1881 -lion -traded -photographs -writes -craig -##fa -trials -generated -beth -noble -debt -percentage -yorkshire -erected -ss -viewed -grades -confidence -ceased -islam -telephone -retail -##ible -chile -m² -roberts -sixteen -##ich -commented -hampshire -innocent -dual -pounds -checked -regulations -afghanistan -sung -rico -liberty -assets -bigger -options -angels -relegated -tribute -wells -attending -leaf -##yan -butler -romanian -forum -monthly -lisa -patterns -gmina -##tory -madison -hurricane -rev -##ians -bristol -##ula -elite -valuable -disaster -democracy -awareness -germans -freyja -##ins -loop -absolutely -paying -populations -maine -sole -prayer -spencer -releases -doorway -bull -##ani -lover -midnight -conclusion -##sson -thirteen -lily -mediterranean -##lt -nhl -proud -sample -##hill -drummer -guinea -##ova -murphy -climb -##ston -instant -attributed -horn -ain -railways -steven -##ao -autumn -ferry -opponent -root -traveling -secured -corridor -stretched -tales -sheet -trinity -cattle -helps -indicates -manhattan -murdered -fitted -1882 -gentle -grandmother -mines -shocked -vegas -produces -##light -caribbean -##ou -belong -continuous -desperate -drunk -historically -trio -waved -raf -dealing -nathan -bat -murmured -interrupted -residing -scientist -pioneer -harold -aaron -##net -delta -attempting -minority -mini -believes -chorus -tend -lots -eyed -indoor -load -shots -updated -jail -##llo -concerning -connecting -wealth -##ved -slaves -arrive -rangers -sufficient -rebuilt -##wick -cardinal -flood -muhammad -whenever -relation -runners -moral -repair -viewers -arriving -revenge -punk -assisted -bath -fairly -breathe -lists -innings -illustrated -whisper -nearest -voters -clinton -ties -ultimate -screamed -beijing -lions -andre -fictional -gathering -comfort -radar -suitable -dismissed -hms -ban -pine -wrist -atmosphere -voivodeship -bid -timber -##ned -##nan -giants -##ane -cameron -recovery -uss -identical -categories -switched -serbia -laughter -noah -ensemble -therapy -peoples -touching -##off -locally -pearl -platforms -everywhere -ballet -tables -lanka -herbert -outdoor -toured -derek -1883 -spaces -contested -swept -1878 -exclusive -slight -connections -##dra -winds -prisoner -collective -bangladesh -tube -publicly -wealthy -thai -##ys -isolated -select -##ric -insisted -pen -fortune -ticket -spotted -reportedly -animation -enforcement -tanks -110 -decides -wider -lowest -owen -##time -nod -hitting -##hn -gregory -furthermore -magazines -fighters -solutions -##ery -pointing -requested -peru -reed -chancellor -knights -mask -worker -eldest -flames -reduction -1860 -volunteers -##tis -reporting -##hl -wire -advisory -endemic -origins -settlers -pursue -knock -consumer -1876 -eu -compound -creatures -mansion -sentenced -ivan -deployed -guitars -frowned -involves -mechanism -kilometers -perspective -shops -maps -terminus -duncan -alien -fist -bridges -##pers -heroes -fed -derby -swallowed -##ros -patent -sara -illness -characterized -adventures -slide -hawaii -jurisdiction -##op -organised -##side -adelaide -walks -biology -se -##ties -rogers -swing -tightly -boundaries -##rie -prepare -implementation -stolen -##sha -certified -colombia -edwards -garage -##mm -recalled -##ball -rage -harm -nigeria -breast -##ren -furniture -pupils -settle -##lus -cuba -balls -client -alaska -21st -linear -thrust -celebration -latino -genetic -terror -##cia -##ening -lightning -fee -witness -lodge -establishing -skull -##ique -earning -hood -##ei -rebellion -wang -sporting -warned -missile -devoted -activist -porch -worship -fourteen -package -1871 -decorated -##shire -housed -##ock -chess -sailed -doctors -oscar -joan -treat -garcia -harbour -jeremy -##ire -traditions -dominant -jacques -##gon -##wan -relocated -1879 -amendment -sized -companion -simultaneously -volleyball -spun -acre -increases -stopping -loves -belongs -affect -drafted -tossed -scout -battles -1875 -filming -shoved -munich -tenure -vertical -romance -pc -##cher -argue -##ical -craft -ranging -www -opens -honest -tyler -yesterday -virtual -##let -muslims -reveal -snake -immigrants -radical -screaming -speakers -firing -saving -belonging -ease -lighting -prefecture -blame -farmer -hungry -grows -rubbed -beam -sur -subsidiary -##cha -armenian -sao -dropping -conventional -##fer -microsoft -reply -qualify -spots -1867 -sweat -festivals -##ken -immigration -physician -discover -exposure -sandy -explanation -isaac -implemented -##fish -hart -initiated -connect -stakes -presents -heights -householder -pleased -tourist -regardless -slip -closest -##ction -surely -sultan -brings -riley -preparation -aboard -slammed -baptist -experiment -ongoing -interstate -organic -playoffs -##ika -1877 -130 -##tar -hindu -error -tours -tier -plenty -arrangements -talks -trapped -excited -sank -ho -athens -1872 -denver -welfare -suburb -athletes -trick -diverse -belly -exclusively -yelled -1868 -##med -conversion -##ette -1874 -internationally -computers -conductor -abilities -sensitive -hello -dispute -measured -globe -rocket -prices -amsterdam -flights -tigers -inn -municipalities -emotion -references -3d -##mus -explains -airlines -manufactured -pm -archaeological -1873 -interpretation -devon -comment -##ites -settlements -kissing -absolute -improvement -suite -impressed -barcelona -sullivan -jefferson -towers -jesse -julie -##tin -##lu -grandson -hi -gauge -regard -rings -interviews -trace -raymond -thumb -departments -burns -serial -bulgarian -scores -demonstrated -##ix -1866 -kyle -alberta -underneath -romanized -##ward -relieved -acquisition -phrase -cliff -reveals -han -cuts -merger -custom -##dar -nee -gilbert -graduation -##nts -assessment -cafe -difficulty -demands -swung -democrat -jennifer -commons -1940s -grove -##yo -completing -focuses -sum -substitute -bearing -stretch -reception -##py -reflected -essentially -destination -pairs -##ched -survival -resource -##bach -promoting -doubles -messages -tear -##down -##fully -parade -florence -harvey -incumbent -partial -framework -900 -pedro -frozen -procedure -olivia -controls -##mic -shelter -personally -temperatures -##od -brisbane -tested -sits -marble -comprehensive -oxygen -leonard -##kov -inaugural -iranian -referring -quarters -attitude -##ivity -mainstream -lined -mars -dakota -norfolk -unsuccessful -##° -explosion -helicopter -congressional -##sing -inspector -bitch -seal -departed -divine -##ters -coaching -examination -punishment -manufacturer -sink -columns -unincorporated -signals -nevada -squeezed -dylan -dining -photos -martial -manuel -eighteen -elevator -brushed -plates -ministers -ivy -congregation -##len -slept -specialized -taxes -curve -restricted -negotiations -likes -statistical -arnold -inspiration -execution -bold -intermediate -significance -margin -ruler -wheels -gothic -intellectual -dependent -listened -eligible -buses -widow -syria -earn -cincinnati -collapsed -recipient -secrets -accessible -philippine -maritime -goddess -clerk -surrender -breaks -playoff -database -##ified -##lon -ideal -beetle -aspect -soap -regulation -strings -expand -anglo -shorter -crosses -retreat -tough -coins -wallace -directions -pressing -##oon -shipping -locomotives -comparison -topics -nephew -##mes -distinction -honors -travelled -sierra -ibn -##over -fortress -sa -recognised -carved -1869 -clients -##dan -intent -##mar -coaches -describing -bread -##ington -beaten -northwestern -##ona -merit -youtube -collapse -challenges -em -historians -objective -submitted -virus -attacking -drake -assume -##ere -diseases -marc -stem -leeds -##cus -##ab -farming -glasses -##lock -visits -nowhere -fellowship -relevant -carries -restaurants -experiments -101 -constantly -bases -targets -shah -tenth -opponents -verse -territorial -##ira -writings -corruption -##hs -instruction -inherited -reverse -emphasis -##vic -employee -arch -keeps -rabbi -watson -payment -uh -##ala -nancy -##tre -venice -fastest -sexy -banned -adrian -properly -ruth -touchdown -dollar -boards -metre -circles -edges -favour -comments -ok -travels -liberation -scattered -firmly -##ular -holland -permitted -diesel -kenya -den -originated -##ral -demons -resumed -dragged -rider -##rus -servant -blinked -extend -torn -##ias -##sey -input -meal -everybody -cylinder -kinds -camps -##fe -bullet -logic -##wn -croatian -evolved -healthy -fool -chocolate -wise -preserve -pradesh -##ess -respective -1850 -##ew -chicken -artificial -gross -corresponding -convicted -cage -caroline -dialogue -##dor -narrative -stranger -mario -br -christianity -failing -trent -commanding -buddhist -1848 -maurice -focusing -yale -bike -altitude -##ering -mouse -revised -##sley -veteran -##ig -pulls -theology -crashed -campaigns -legion -##ability -drag -excellence -customer -cancelled -intensity -excuse -##lar -liga -participating -contributing -printing -##burn -variable -##rk -curious -bin -legacy -renaissance -##my -symptoms -binding -vocalist -dancer -##nie -grammar -gospel -democrats -ya -enters -sc -diplomatic -hitler -##ser -clouds -mathematical -quit -defended -oriented -##heim -fundamental -hardware -impressive -equally -convince -confederate -guilt -chuck -sliding -##ware -magnetic -narrowed -petersburg -bulgaria -otto -phd -skill -##ama -reader -hopes -pitcher -reservoir -hearts -automatically -expecting -mysterious -bennett -extensively -imagined -seeds -monitor -fix -##ative -journalism -struggling -signature -ranch -encounter -photographer -observation -protests -##pin -influences -##hr -calendar -##all -cruz -croatia -locomotive -hughes -naturally -shakespeare -basement -hook -uncredited -faded -theories -approaches -dare -phillips -filling -fury -obama -##ain -efficient -arc -deliver -min -raid -breeding -inducted -leagues -efficiency -axis -montana -eagles -##ked -supplied -instructions -karen -picking -indicating -trap -anchor -practically -christians -tomb -vary -occasional -electronics -lords -readers -newcastle -faint -innovation -collect -situations -engagement -160 -claude -mixture -##feld -peer -tissue -logo -lean -##ration -°f -floors -##ven -architects -reducing -##our -##ments -rope -1859 -ottawa -##har -samples -banking -declaration -proteins -resignation -francois -saudi -advocate -exhibited -armor -twins -divorce -##ras -abraham -reviewed -jo -temporarily -matrix -physically -pulse -curled -##ena -difficulties -bengal -usage -##ban -annie -riders -certificate -##pi -holes -warsaw -distinctive -jessica -##mon -mutual -1857 -customs -circular -eugene -removal -loaded -mere -vulnerable -depicted -generations -dame -heir -enormous -lightly -climbing -pitched -lessons -pilots -nepal -ram -google -preparing -brad -louise -renowned -##₂ -liam -##ably -plaza -shaw -sophie -brilliant -bills -##bar -##nik -fucking -mainland -server -pleasant -seized -veterans -jerked -fail -beta -brush -radiation -stored -warmth -southeastern -nate -sin -raced -berkeley -joke -athlete -designation -trunk -##low -roland -qualification -archives -heels -artwork -receives -judicial -reserves -##bed -woke -installation -abu -floating -fake -lesser -excitement -interface -concentrated -addressed -characteristic -amanda -saxophone -monk -auto -##bus -releasing -egg -dies -interaction -defender -ce -outbreak -glory -loving -##bert -sequel -consciousness -http -awake -ski -enrolled -##ress -handling -rookie -brow -somebody -biography -warfare -amounts -contracts -presentation -fabric -dissolved -challenged -meter -psychological -lt -elevated -rally -accurate -##tha -hospitals -undergraduate -specialist -venezuela -exhibit -shed -nursing -protestant -fluid -structural -footage -jared -consistent -prey -##ska -succession -reflect -exile -lebanon -wiped -suspect -shanghai -resting -integration -preservation -marvel -variant -pirates -sheep -rounded -capita -sailing -colonies -manuscript -deemed -variations -clarke -functional -emerging -boxing -relaxed -curse -azerbaijan -heavyweight -nickname -editorial -rang -grid -tightened -earthquake -flashed -miguel -rushing -##ches -improvements -boxes -brooks -180 -consumption -molecular -felix -societies -repeatedly -variation -aids -civic -graphics -professionals -realm -autonomous -receiver -delayed -workshop -militia -chairs -trump -canyon -##point -harsh -extending -lovely -happiness -##jan -stake -eyebrows -embassy -wellington -hannah -##ella -sony -corners -bishops -swear -cloth -contents -xi -namely -commenced -1854 -stanford -nashville -courage -graphic -commitment -garrison -##bin -hamlet -clearing -rebels -attraction -literacy -cooking -ruins -temples -jenny -humanity -celebrate -hasn -freight -sixty -rebel -bastard -##art -newton -##ada -deer -##ges -##ching -smiles -delaware -singers -##ets -approaching -assists -flame -##ph -boulevard -barrel -planted -##ome -pursuit -##sia -consequences -posts -shallow -invitation -rode -depot -ernest -kane -rod -concepts -preston -topic -chambers -striking -blast -arrives -descendants -montgomery -ranges -worlds -##lay -##ari -span -chaos -praise -##ag -fewer -1855 -sanctuary -mud -fbi -##ions -programmes -maintaining -unity -harper -bore -handsome -closure -tournaments -thunder -nebraska -linda -facade -puts -satisfied -argentine -dale -cork -dome -panama -##yl -1858 -tasks -experts -##ates -feeding -equation -##las -##ida -##tu -engage -bryan -##ax -um -quartet -melody -disbanded -sheffield -blocked -gasped -delay -kisses -maggie -connects -##non -sts -poured -creator -publishers -##we -guided -ellis -extinct -hug -gaining -##ord -complicated -##bility -poll -clenched -investigate -##use -thereby -quantum -spine -cdp -humor -kills -administered -semifinals -##du -encountered -ignore -##bu -commentary -##maker -bother -roosevelt -140 -plains -halfway -flowing -cultures -crack -imprisoned -neighboring -airline -##ses -##view -##mate -##ec -gather -wolves -marathon -transformed -##ill -cruise -organisations -carol -punch -exhibitions -numbered -alarm -ratings -daddy -silently -##stein -queens -colours -impression -guidance -liu -tactical -##rat -marshal -della -arrow -##ings -rested -feared -tender -owns -bitter -advisor -escort -##ides -spare -farms -grants -##ene -dragons -encourage -colleagues -cameras -##und -sucked -pile -spirits -prague -statements -suspension -landmark -fence -torture -recreation -bags -permanently -survivors -pond -spy -predecessor -bombing -coup -##og -protecting -transformation -glow -##lands -##book -dug -priests -andrea -feat -barn -jumping -##chen -##ologist -##con -casualties -stern -auckland -pipe -serie -revealing -ba -##bel -trevor -mercy -spectrum -yang -consist -governing -collaborated -possessed -epic -comprises -blew -shane -##ack -lopez -honored -magical -sacrifice -judgment -perceived -hammer -mtv -baronet -tune -das -missionary -sheets -350 -neutral -oral -threatening -attractive -shade -aims -seminary -##master -estates -1856 -michel -wounds -refugees -manufacturers -##nic -mercury -syndrome -porter -##iya -##din -hamburg -identification -upstairs -purse -widened -pause -cared -breathed -affiliate -santiago -prevented -celtic -fisher -125 -recruited -byzantine -reconstruction -farther -##mp -diet -sake -au -spite -sensation -##ert -blank -separation -105 -##hon -vladimir -armies -anime -##lie -accommodate -orbit -cult -sofia -archive -##ify -##box -founders -sustained -disorder -honours -northeastern -mia -crops -violet -threats -blanket -fires -canton -followers -southwestern -prototype -voyage -assignment -altered -moderate -protocol -pistol -##eo -questioned -brass -lifting -1852 -math -authored -##ual -doug -dimensional -dynamic -##san -1851 -pronounced -grateful -quest -uncomfortable -boom -presidency -stevens -relating -politicians -chen -barrier -quinn -diana -mosque -tribal -cheese -palmer -portions -sometime -chester -treasure -wu -bend -download -millions -reforms -registration -##osa -consequently -monitoring -ate -preliminary -brandon -invented -ps -eaten -exterior -intervention -ports -documented -log -displays -lecture -sally -favourite -##itz -vermont -lo -invisible -isle -breed -##ator -journalists -relay -speaks -backward -explore -midfielder -actively -stefan -procedures -cannon -blond -kenneth -centered -servants -chains -libraries -malcolm -essex -henri -slavery -##hal -facts -fairy -coached -cassie -cats -washed -cop -##fi -announcement -item -2000s -vinyl -activated -marco -frontier -growled -curriculum -##das -loyal -accomplished -leslie -ritual -kenny -##00 -vii -napoleon -hollow -hybrid -jungle -stationed -friedrich -counted -##ulated -platinum -theatrical -seated -col -rubber -glen -1840 -diversity -healing -extends -id -provisions -administrator -columbus -##oe -tributary -te -assured -org -##uous -prestigious -examined -lectures -grammy -ronald -associations -bailey -allan -essays -flute -believing -consultant -proceedings -travelling -1853 -kit -kerala -yugoslavia -buddy -methodist -##ith -burial -centres -batman -##nda -discontinued -bo -dock -stockholm -lungs -severely -##nk -citing -manga -##ugh -steal -mumbai -iraqi -robot -celebrity -bride -broadcasts -abolished -pot -joel -overhead -franz -packed -reconnaissance -johann -acknowledged -introduce -handled -doctorate -developments -drinks -alley -palestine -##nis -##aki -proceeded -recover -bradley -grain -patch -afford -infection -nationalist -legendary -##ath -interchange -virtually -gen -gravity -exploration -amber -vital -wishes -powell -doctrine -elbow -screenplay -##bird -contribute -indonesian -pet -creates -##com -enzyme -kylie -discipline -drops -manila -hunger -##ien -layers -suffer -fever -bits -monica -keyboard -manages -##hood -searched -appeals -##bad -testament -grande -reid -##war -beliefs -congo -##ification -##dia -si -requiring -##via -casey -1849 -regret -streak -rape -depends -syrian -sprint -pound -tourists -upcoming -pub -##xi -tense -##els -practiced -echo -nationwide -guild -motorcycle -liz -##zar -chiefs -desired -elena -bye -precious -absorbed -relatives -booth -pianist -##mal -citizenship -exhausted -wilhelm -##ceae -##hed -noting -quarterback -urge -hectares -##gue -ace -holly -##tal -blonde -davies -parked -sustainable -stepping -twentieth -airfield -galaxy -nest -chip -##nell -tan -shaft -paulo -requirement -##zy -paradise -tobacco -trans -renewed -vietnamese -##cker -##ju -suggesting -catching -holmes -enjoying -md -trips -colt -holder -butterfly -nerve -reformed -cherry -bowling -trailer -carriage -goodbye -appreciate -toy -joshua -interactive -enabled -involve -##kan -collar -determination -bunch -facebook -recall -shorts -superintendent -episcopal -frustration -giovanni -nineteenth -laser -privately -array -circulation -##ovic -armstrong -deals -painful -permit -discrimination -##wi -aires -retiring -cottage -ni -##sta -horizon -ellen -jamaica -ripped -fernando -chapters -playstation -patron -lecturer -navigation -behaviour -genes -georgian -export -solomon -rivals -swift -seventeen -rodriguez -princeton -independently -sox -1847 -arguing -entity -casting -hank -criteria -oakland -geographic -milwaukee -reflection -expanding -conquest -dubbed -##tv -halt -brave -brunswick -doi -arched -curtis -divorced -predominantly -somerset -streams -ugly -zoo -horrible -curved -buenos -fierce -dictionary -vector -theological -unions -handful -stability -chan -punjab -segments -##lly -altar -ignoring -gesture -monsters -pastor -##stone -thighs -unexpected -operators -abruptly -coin -compiled -associates -improving -migration -pin -##ose -compact -collegiate -reserved -##urs -quarterfinals -roster -restore -assembled -hurry -oval -##cies -1846 -flags -martha -##del -victories -sharply -##rated -argues -deadly -neo -drawings -symbols -performer -##iel -griffin -restrictions -editing -andrews -java -journals -arabia -compositions -dee -pierce -removing -hindi -casino -runway -civilians -minds -nasa -hotels -##zation -refuge -rent -retain -potentially -conferences -suburban -conducting -##tto -##tions -##tle -descended -massacre -##cal -ammunition -terrain -fork -souls -counts -chelsea -durham -drives -cab -##bank -perth -realizing -palestinian -finn -simpson -##dal -betty -##ule -moreover -particles -cardinals -tent -evaluation -extraordinary -##oid -inscription -##works -wednesday -chloe -maintains -panels -ashley -trucks -##nation -cluster -sunlight -strikes -zhang -##wing -dialect -canon -##ap -tucked -##ws -collecting -##mas -##can -##sville -maker -quoted -evan -franco -aria -buying -cleaning -eva -closet -provision -apollo -clinic -rat -##ez -necessarily -ac -##gle -##ising -venues -flipped -cent -spreading -trustees -checking -authorized -##sco -disappointed -##ado -notion -duration -trumpet -hesitated -topped -brussels -rolls -theoretical -hint -define -aggressive -repeat -wash -peaceful -optical -width -allegedly -mcdonald -strict -copyright -##illa -investors -mar -jam -witnesses -sounding -miranda -michelle -privacy -hugo -harmony -##pp -valid -lynn -glared -nina -102 -headquartered -diving -boarding -gibson -##ncy -albanian -marsh -routine -dealt -enhanced -er -intelligent -substance -targeted -enlisted -discovers -spinning -observations -pissed -smoking -rebecca -capitol -visa -varied -costume -seemingly -indies -compensation -surgeon -thursday -arsenal -westminster -suburbs -rid -anglican -##ridge -knots -foods -alumni -lighter -fraser -whoever -portal -scandal -##ray -gavin -advised -instructor -flooding -terrorist -##ale -teenage -interim -senses -duck -teen -thesis -abby -eager -overcome -##ile -newport -glenn -rises -shame -##cc -prompted -priority -forgot -bomber -nicolas -protective -360 -cartoon -katherine -breeze -lonely -trusted -henderson -richardson -relax -banner -candy -palms -remarkable -##rio -legends -cricketer -essay -ordained -edmund -rifles -trigger -##uri -##away -sail -alert -1830 -audiences -penn -sussex -siblings -pursued -indianapolis -resist -rosa -consequence -succeed -avoided -1845 -##ulation -inland -##tie -##nna -counsel -profession -chronicle -hurried -##una -eyebrow -eventual -bleeding -innovative -cure -##dom -committees -accounting -con -scope -hardy -heather -tenor -gut -herald -codes -tore -scales -wagon -##oo -luxury -tin -prefer -fountain -triangle -bonds -darling -convoy -dried -traced -beings -troy -accidentally -slam -findings -smelled -joey -lawyers -outcome -steep -bosnia -configuration -shifting -toll -brook -performers -lobby -philosophical -construct -shrine -aggregate -boot -cox -phenomenon -savage -insane -solely -reynolds -lifestyle -##ima -nationally -holdings -consideration -enable -edgar -mo -mama -##tein -fights -relegation -chances -atomic -hub -conjunction -awkward -reactions -currency -finale -kumar -underwent -steering -elaborate -gifts -comprising -melissa -veins -reasonable -sunshine -chi -solve -trails -inhabited -elimination -ethics -huh -ana -molly -consent -apartments -layout -marines -##ces -hunters -bulk -##oma -hometown -##wall -##mont -cracked -reads -neighbouring -withdrawn -admission -wingspan -damned -anthology -lancashire -brands -batting -forgive -cuban -awful -##lyn -104 -dimensions -imagination -##ade -dante -##ship -tracking -desperately -goalkeeper -##yne -groaned -workshops -confident -burton -gerald -milton -circus -uncertain -slope -copenhagen -sophia -fog -philosopher -portraits -accent -cycling -varying -gripped -larvae -garrett -specified -scotia -mature -luther -kurt -rap -##kes -aerial -750 -ferdinand -heated -es -transported -##shan -safely -nonetheless -##orn -##gal -motors -demanding -##sburg -startled -##brook -ally -generate -caps -ghana -stained -demo -mentions -beds -ap -afterward -diary -##bling -utility -##iro -richards -1837 -conspiracy -conscious -shining -footsteps -observer -cyprus -urged -loyalty -developer -probability -olive -upgraded -gym -miracle -insects -graves -1844 -ourselves -hydrogen -amazon -katie -tickets -poets -##pm -planes -##pan -prevention -witnessed -dense -jin -randy -tang -warehouse -monroe -bang -archived -elderly -investigations -alec -granite -mineral -conflicts -controlling -aboriginal -carlo -##zu -mechanics -stan -stark -rhode -skirt -est -##berry -bombs -respected -##horn -imposed -limestone -deny -nominee -memphis -grabbing -disabled -##als -amusement -aa -frankfurt -corn -referendum -varies -slowed -disk -firms -unconscious -incredible -clue -sue -##zhou -twist -##cio -joins -idaho -chad -developers -computing -destroyer -103 -mortal -tucker -kingston -choices -yu -carson -1800 -os -whitney -geneva -pretend -dimension -staged -plateau -maya -##une -freestyle -##bc -rovers -hiv -##ids -tristan -classroom -prospect -##hus -honestly -diploma -lied -thermal -auxiliary -feast -unlikely -iata -##tel -morocco -pounding -treasury -lithuania -considerably -1841 -dish -1812 -geological -matching -stumbled -destroying -marched -brien -advances -cake -nicole -belle -settling -measuring -directing -##mie -tuesday -bassist -capabilities -stunned -fraud -torpedo -##list -##phone -anton -wisdom -surveillance -ruined -##ulate -lawsuit -healthcare -theorem -halls -trend -aka -horizontal -dozens -acquire -lasting -swim -hawk -gorgeous -fees -vicinity -decrease -adoption -tactics -##ography -pakistani -##ole -draws -##hall -willie -burke -heath -algorithm -integral -powder -elliott -brigadier -jackie -tate -varieties -darker -##cho -lately -cigarette -specimens -adds -##ree -##ensis -##inger -exploded -finalist -cia -murders -wilderness -arguments -nicknamed -acceptance -onwards -manufacture -robertson -jets -tampa -enterprises -blog -loudly -composers -nominations -1838 -ai -malta -inquiry -automobile -hosting -viii -rays -tilted -grief -museums -strategies -furious -euro -equality -cohen -poison -surrey -wireless -governed -ridiculous -moses -##esh -##room -vanished -##ito -barnes -attract -morrison -istanbul -##iness -absent -rotation -petition -janet -##logical -satisfaction -custody -deliberately -observatory -comedian -surfaces -pinyin -novelist -strictly -canterbury -oslo -monks -embrace -ibm -jealous -photograph -continent -dorothy -marina -doc -excess -holden -allegations -explaining -stack -avoiding -lance -storyline -majesty -poorly -spike -dos -bradford -raven -travis -classics -proven -voltage -pillow -fists -butt -1842 -interpreted -##car -1839 -gage -telegraph -lens -promising -expelled -casual -collector -zones -##min -silly -nintendo -##kh -##bra -downstairs -chef -suspicious -afl -flies -vacant -uganda -pregnancy -condemned -lutheran -estimates -cheap -decree -saxon -proximity -stripped -idiot -deposits -contrary -presenter -magnus -glacier -im -offense -edwin -##ori -upright -##long -bolt -##ois -toss -geographical -##izes -environments -delicate -marking -abstract -xavier -nails -windsor -plantation -occurring -equity -saskatchewan -fears -drifted -sequences -vegetation -revolt -##stic -1843 -sooner -fusion -opposing -nato -skating -1836 -secretly -ruin -lease -##oc -edit -##nne -flora -anxiety -ruby -##ological -##mia -tel -bout -taxi -emmy -frost -rainbow -compounds -foundations -rainfall -assassination -nightmare -dominican -##win -achievements -deserve -orlando -intact -armenia -##nte -calgary -valentine -106 -marion -proclaimed -theodore -bells -courtyard -thigh -gonzalez -console -troop -minimal -monte -everyday -##ence -##if -supporter -terrorism -buck -openly -presbyterian -activists -carpet -##iers -rubbing -uprising -##yi -cute -conceived -legally -##cht -millennium -cello -velocity -ji -rescued -cardiff -1835 -rex -concentrate -senators -beard -rendered -glowing -battalions -scouts -competitors -sculptor -catalogue -arctic -ion -raja -bicycle -wow -glancing -lawn -##woman -gentleman -lighthouse -publish -predicted -calculated -##val -variants -##gne -strain -##ui -winston -deceased -##nus -touchdowns -brady -caleb -sinking -echoed -crush -hon -blessed -protagonist -hayes -endangered -magnitude -editors -##tine -estimate -responsibilities -##mel -backup -laying -consumed -sealed -zurich -lovers -frustrated -##eau -ahmed -kicking -mit -treasurer -1832 -biblical -refuse -terrified -pump -agrees -genuine -imprisonment -refuses -plymouth -##hen -lou -##nen -tara -trembling -antarctic -ton -learns -##tas -crap -crucial -faction -atop -##borough -wrap -lancaster -odds -hopkins -erik -lyon -##eon -bros -##ode -snap -locality -tips -empress -crowned -cal -acclaimed -chuckled -##ory -clara -sends -mild -towel -##fl -##day -##а -wishing -assuming -interviewed -##bal -##die -interactions -eden -cups -helena -##lf -indie -beck -##fire -batteries -filipino -wizard -parted -##lam -traces -##born -rows -idol -albany -delegates -##ees -##sar -discussions -##ex -notre -instructed -belgrade -highways -suggestion -lauren -possess -orientation -alexandria -abdul -beats -salary -reunion -ludwig -alright -wagner -intimate -pockets -slovenia -hugged -brighton -merchants -cruel -stole -trek -slopes -repairs -enrollment -politically -underlying -promotional -counting -boeing -##bb -isabella -naming -##и -keen -bacteria -listing -separately -belfast -ussr -450 -lithuanian -anybody -ribs -sphere -martinez -cock -embarrassed -proposals -fragments -nationals -##fs -##wski -premises -fin -1500 -alpine -matched -freely -bounded -jace -sleeve -##af -gaming -pier -populated -evident -##like -frances -flooded -##dle -frightened -pour -trainer -framed -visitor -challenging -pig -wickets -##fold -infected -email -##pes -arose -##aw -reward -ecuador -oblast -vale -ch -shuttle -##usa -bach -rankings -forbidden -cornwall -accordance -salem -consumers -bruno -fantastic -toes -machinery -resolved -julius -remembering -propaganda -iceland -bombardment -tide -contacts -wives -##rah -concerto -macdonald -albania -implement -daisy -tapped -sudan -helmet -angela -mistress -##lic -crop -sunk -finest -##craft -hostile -##ute -##tsu -boxer -fr -paths -adjusted -habit -ballot -supervision -soprano -##zen -bullets -wicked -sunset -regiments -disappear -lamp -performs -app -##gia -##oa -rabbit -digging -incidents -entries -##cion -dishes -##oi -introducing -##ati -##fied -freshman -slot -jill -tackles -baroque -backs -##iest -lone -sponsor -destiny -altogether -convert -##aro -consensus -shapes -demonstration -basically -feminist -auction -artifacts -##bing -strongest -twitter -halifax -2019 -allmusic -mighty -smallest -precise -alexandra -viola -##los -##ille -manuscripts -##illo -dancers -ari -managers -monuments -blades -barracks -springfield -maiden -consolidated -electron -##end -berry -airing -wheat -nobel -inclusion -blair -payments -geography -bee -cc -eleanor -react -##hurst -afc -manitoba -##yu -su -lineup -fitness -recreational -investments -airborne -disappointment -##dis -edmonton -viewing -##row -renovation -##cast -infant -bankruptcy -roses -aftermath -pavilion -##yer -carpenter -withdrawal -ladder -##hy -discussing -popped -reliable -agreements -rochester -##abad -curves -bombers -220 -rao -reverend -decreased -choosing -107 -stiff -consulting -naples -crawford -tracy -ka -ribbon -cops -##lee -crushed -deciding -unified -teenager -accepting -flagship -explorer -poles -sanchez -inspection -revived -skilled -induced -exchanged -flee -locals -tragedy -swallow -loading -hanna -demonstrate -##ela -salvador -flown -contestants -civilization -##ines -wanna -rhodes -fletcher -hector -knocking -considers -##ough -nash -mechanisms -sensed -mentally -walt -unclear -##eus -renovated -madame -##cks -crews -governmental -##hin -undertaken -monkey -##ben -##ato -fatal -armored -copa -caves -governance -grasp -perception -certification -froze -damp -tugged -wyoming -##rg -##ero -newman -##lor -nerves -curiosity -graph -115 -##ami -withdraw -tunnels -dull -meredith -moss -exhibits -neighbors -communicate -accuracy -explored -raiders -republicans -secular -kat -superman -penny -criticised -##tch -freed -update -conviction -wade -ham -likewise -delegation -gotta -doll -promises -technological -myth -nationality -resolve -convent -##mark -sharon -dig -sip -coordinator -entrepreneur -fold -##dine -capability -councillor -synonym -blown -swan -cursed -1815 -jonas -haired -sofa -canvas -keeper -rivalry -##hart -rapper -speedway -swords -postal -maxwell -estonia -potter -recurring -##nn -##ave -errors -##oni -cognitive -1834 -##² -claws -nadu -roberto -bce -wrestler -ellie -##ations -infinite -ink -##tia -presumably -finite -staircase -108 -noel -patricia -nacional -##cation -chill -eternal -tu -preventing -prussia -fossil -limbs -##logist -ernst -frog -perez -rene -##ace -pizza -prussian -##ios -##vy -molecules -regulatory -answering -opinions -sworn -lengths -supposedly -hypothesis -upward -habitats -seating -ancestors -drank -yield -hd -synthesis -researcher -modest -##var -mothers -peered -voluntary -homeland -##the -acclaim -##igan -static -valve -luxembourg -alto -carroll -fe -receptor -norton -ambulance -##tian -johnston -catholics -depicting -jointly -elephant -gloria -mentor -badge -ahmad -distinguish -remarked -councils -precisely -allison -advancing -detection -crowded -##10 -cooperative -ankle -mercedes -dagger -surrendered -pollution -commit -subway -jeffrey -lesson -sculptures -provider -##fication -membrane -timothy -rectangular -fiscal -heating -teammate -basket -particle -anonymous -deployment -##ple -missiles -courthouse -proportion -shoe -sec -##ller -complaints -forbes -blacks -abandon -remind -sizes -overwhelming -autobiography -natalie -##awa -risks -contestant -countryside -babies -scorer -invaded -enclosed -proceed -hurling -disorders -##cu -reflecting -continuously -cruiser -graduates -freeway -investigated -ore -deserved -maid -blocking -phillip -jorge -shakes -dove -mann -variables -lacked -burden -accompanying -que -consistently -organizing -provisional -complained -endless -##rm -tubes -juice -georges -krishna -mick -labels -thriller -##uch -laps -arcade -sage -snail -##table -shannon -fi -laurence -seoul -vacation -presenting -hire -churchill -surprisingly -prohibited -savannah -technically -##oli -170 -##lessly -testimony -suited -speeds -toys -romans -mlb -flowering -measurement -talented -kay -settings -charleston -expectations -shattered -achieving -triumph -ceremonies -portsmouth -lanes -mandatory -loser -stretching -cologne -realizes -seventy -cornell -careers -webb -##ulating -americas -budapest -ava -suspicion -##ison -yo -conrad -##hai -sterling -jessie -rector -##az -1831 -transform -organize -loans -christine -volcanic -warrant -slender -summers -subfamily -newer -danced -dynamics -rhine -proceeds -heinrich -gastropod -commands -sings -facilitate -easter -ra -positioned -responses -expense -fruits -yanked -imported -25th -velvet -vic -primitive -tribune -baldwin -neighbourhood -donna -rip -hay -pr -##uro -1814 -espn -welcomed -##aria -qualifier -glare -highland -timing -##cted -shells -eased -geometry -louder -exciting -slovakia -##sion -##iz -##lot -savings -prairie -##ques -marching -rafael -tonnes -##lled -curtain -preceding -shy -heal -greene -worthy -##pot -detachment -bury -sherman -##eck -reinforced -seeks -bottles -contracted -duchess -outfit -walsh -##sc -mickey -##ase -geoffrey -archer -squeeze -dawson -eliminate -invention -##enberg -neal -##eth -stance -dealer -coral -maple -retire -polo -simplified -##ht -1833 -hid -watts -backwards -jules -##oke -genesis -mt -frames -rebounds -burma -woodland -moist -santos -whispers -drained -subspecies -##aa -streaming -ulster -burnt -correspondence -maternal -gerard -denis -stealing -##load -genius -duchy -##oria -inaugurated -momentum -suits -placement -sovereign -clause -thames -##hara -confederation -reservation -sketch -yankees -lets -rotten -charm -hal -verses -ultra -commercially -dot -salon -citation -adopt -winnipeg -mist -allocated -cairo -##boy -jenkins -interference -objectives -##wind -1820 -portfolio -armoured -sectors -##eh -initiatives -##world -integrity -exercises -robe -tap -ab -gazed -##tones -distracted -rulers -111 -favorable -jerome -tended -cart -factories -##eri -diplomat -valued -gravel -charitable -##try -calvin -exploring -chang -shepherd -terrace -pdf -pupil -##ural -reflects -ups -##rch -governors -shelf -depths -##nberg -trailed -crest -tackle -##nian -##ats -hatred -##kai -clare -makers -ethiopia -longtime -detected -embedded -lacking -slapped -rely -thomson -anticipation -iso -morton -successive -agnes -screenwriter -straightened -philippe -playwright -haunted -licence -iris -intentions -sutton -112 -logical -correctly -##weight -branded -licked -tipped -silva -ricky -narrator -requests -##ents -greeted -supernatural -cow -##wald -lung -refusing -employer -strait -gaelic -liner -##piece -zoe -sabha -##mba -driveway -harvest -prints -bates -reluctantly -threshold -algebra -ira -wherever -coupled -240 -assumption -picks -##air -designers -raids -gentlemen -##ean -roller -blowing -leipzig -locks -screw -dressing -strand -##lings -scar -dwarf -depicts -##nu -nods -##mine -differ -boris -##eur -yuan -flip -##gie -mob -invested -questioning -applying -##ture -shout -##sel -gameplay -blamed -illustrations -bothered -weakness -rehabilitation -##of -##zes -envelope -rumors -miners -leicester -subtle -kerry -##ico -ferguson -##fu -premiership -ne -##cat -bengali -prof -catches -remnants -dana -##rily -shouting -presidents -baltic -ought -ghosts -dances -sailors -shirley -fancy -dominic -##bie -madonna -##rick -bark -buttons -gymnasium -ashes -liver -toby -oath -providence -doyle -evangelical -nixon -cement -carnegie -embarked -hatch -surroundings -guarantee -needing -pirate -essence -##bee -filter -crane -hammond -projected -immune -percy -twelfth -##ult -regent -doctoral -damon -mikhail -##ichi -lu -critically -elect -realised -abortion -acute -screening -mythology -steadily -##fc -frown -nottingham -kirk -wa -minneapolis -##rra -module -algeria -mc -nautical -encounters -surprising -statues -availability -shirts -pie -alma -brows -munster -mack -soup -crater -tornado -sanskrit -cedar -explosive -bordered -dixon -planets -stamp -exam -happily -##bble -carriers -kidnapped -##vis -accommodation -emigrated -##met -knockout -correspondent -violation -profits -peaks -lang -specimen -agenda -ancestry -pottery -spelling -equations -obtaining -ki -linking -1825 -debris -asylum -##20 -buddhism -teddy -##ants -gazette -##nger -##sse -dental -eligibility -utc -fathers -averaged -zimbabwe -francesco -coloured -hissed -translator -lynch -mandate -humanities -mackenzie -uniforms -lin -##iana -##gio -asset -mhz -fitting -samantha -genera -wei -rim -beloved -shark -riot -entities -expressions -indo -carmen -slipping -owing -abbot -neighbor -sidney -##av -rats -recommendations -encouraging -squadrons -anticipated -commanders -conquered -##oto -donations -diagnosed -##mond -divide -##iva -guessed -decoration -vernon -auditorium -revelation -conversations -##kers -##power -herzegovina -dash -alike -protested -lateral -herman -accredited -mg -##gent -freeman -mel -fiji -crow -crimson -##rine -livestock -##pped -humanitarian -bored -oz -whip -##lene -##ali -legitimate -alter -grinning -spelled -anxious -oriental -wesley -##nin -##hole -carnival -controller -detect -##ssa -bowed -educator -kosovo -macedonia -##sin -occupy -mastering -stephanie -janeiro -para -unaware -nurses -noon -135 -cam -hopefully -ranger -combine -sociology -polar -rica -##eer -neill -##sman -holocaust -##ip -doubled -lust -1828 -109 -decent -cooling -unveiled -##card -1829 -nsw -homer -chapman -meyer -##gin -dive -mae -reagan -expertise -##gled -darwin -brooke -sided -prosecution -investigating -comprised -petroleum -genres -reluctant -differently -trilogy -johns -vegetables -corpse -highlighted -lounge -pension -unsuccessfully -elegant -aided -ivory -beatles -amelia -cain -dubai -sunny -immigrant -babe -click -##nder -underwater -pepper -combining -mumbled -atlas -horns -accessed -ballad -physicians -homeless -gestured -rpm -freak -louisville -corporations -patriots -prizes -rational -warn -modes -decorative -overnight -din -troubled -phantom -##ort -monarch -sheer -##dorf -generals -guidelines -organs -addresses -##zon -enhance -curling -parishes -cord -##kie -linux -caesar -deutsche -bavaria -##bia -coleman -cyclone -##eria -bacon -petty -##yama -##old -hampton -diagnosis -1824 -throws -complexity -rita -disputed -##₃ -pablo -##sch -marketed -trafficking -##ulus -examine -plague -formats -##oh -vault -faithful -##bourne -webster -##ox -highlights -##ient -##ann -phones -vacuum -sandwich -modeling -##gated -bolivia -clergy -qualities -isabel -##nas -##ars -wears -screams -reunited -annoyed -bra -##ancy -##rate -differential -transmitter -tattoo -container -poker -##och -excessive -resides -cowboys -##tum -augustus -trash -providers -statute -retreated -balcony -reversed -void -storey -preceded -masses -leap -laughs -neighborhoods -wards -schemes -falcon -santo -battlefield -pad -ronnie -thread -lesbian -venus -##dian -beg -sandstone -daylight -punched -gwen -analog -stroked -wwe -acceptable -measurements -dec -toxic -##kel -adequate -surgical -economist -parameters -varsity -##sberg -quantity -ella -##chy -##rton -countess -generating -precision -diamonds -expressway -ga -##ı -1821 -uruguay -talents -galleries -expenses -scanned -colleague -outlets -ryder -lucien -##ila -paramount -##bon -syracuse -dim -fangs -gown -sweep -##sie -toyota -missionaries -websites -##nsis -sentences -adviser -val -trademark -spells -##plane -patience -starter -slim -##borg -toe -incredibly -shoots -elliot -nobility -##wyn -cowboy -endorsed -gardner -tendency -persuaded -organisms -emissions -kazakhstan -amused -boring -chips -themed -##hand -llc -constantinople -chasing -systematic -guatemala -borrowed -erin -carey -##hard -highlands -struggles -1810 -##ifying -##ced -wong -exceptions -develops -enlarged -kindergarten -castro -##ern -##rina -leigh -zombie -juvenile -##most -consul -##nar -sailor -hyde -clarence -intensive -pinned -nasty -useless -jung -clayton -stuffed -exceptional -ix -apostolic -230 -transactions -##dge -exempt -swinging -cove -religions -##ash -shields -dairy -bypass -190 -pursuing -bug -joyce -bombay -chassis -southampton -chat -interact -redesignated -##pen -nascar -pray -salmon -rigid -regained -malaysian -grim -publicity -constituted -capturing -toilet -delegate -purely -tray -drift -loosely -striker -weakened -trinidad -mitch -itv -defines -transmitted -ming -scarlet -nodding -fitzgerald -fu -narrowly -sp -tooth -standings -virtue -##₁ -##wara -##cting -chateau -gloves -lid -##nel -hurting -conservatory -##pel -sinclair -reopened -sympathy -nigerian -strode -advocated -optional -chronic -discharge -##rc -suck -compatible -laurel -stella -shi -fails -wage -dodge -128 -informal -sorts -levi -buddha -villagers -##aka -chronicles -heavier -summoned -gateway -3000 -eleventh -jewelry -translations -accordingly -seas -##ency -fiber -pyramid -cubic -dragging -##ista -caring -##ops -android -contacted -lunar -##dt -kai -lisbon -patted -1826 -sacramento -theft -madagascar -subtropical -disputes -ta -holidays -piper -willow -mare -cane -itunes -newfoundland -benny -companions -dong -raj -observe -roar -charming -plaque -tibetan -fossils -enacted -manning -bubble -tina -tanzania -##eda -##hir -funk -swamp -deputies -cloak -ufc -scenario -par -scratch -metals -anthem -guru -engaging -specially -##boat -dialects -nineteen -cecil -duet -disability -messenger -unofficial -##lies -defunct -eds -moonlight -drainage -surname -puzzle -honda -switching -conservatives -mammals -knox -broadcaster -sidewalk -cope -##ried -benson -princes -peterson -##sal -bedford -sharks -eli -wreck -alberto -gasp -archaeology -lgbt -teaches -securities -madness -compromise -waving -coordination -davidson -visions -leased -possibilities -eighty -jun -fernandez -enthusiasm -assassin -sponsorship -reviewer -kingdoms -estonian -laboratories -##fy -##nal -applies -verb -celebrations -##zzo -rowing -lightweight -sadness -submit -mvp -balanced -dude -##vas -explicitly -metric -magnificent -mound -brett -mohammad -mistakes -irregular -##hing -##ass -sanders -betrayed -shipped -surge -##enburg -reporters -termed -georg -pity -verbal -bulls -abbreviated -enabling -appealed -##are -##atic -sicily -sting -heel -sweetheart -bart -spacecraft -brutal -monarchy -##tter -aberdeen -cameo -diane -##ub -survivor -clyde -##aries -complaint -##makers -clarinet -delicious -chilean -karnataka -coordinates -1818 -panties -##rst -pretending -ar -dramatically -kiev -bella -tends -distances -113 -catalog -launching -instances -telecommunications -portable -lindsay -vatican -##eim -angles -aliens -marker -stint -screens -bolton -##rne -judy -wool -benedict -plasma -europa -spark -imaging -filmmaker -swiftly -##een -contributor -##nor -opted -stamps -apologize -financing -butter -gideon -sophisticated -alignment -avery -chemicals -yearly -speculation -prominence -professionally -##ils -immortal -institutional -inception -wrists -identifying -tribunal -derives -gains -##wo -papal -preference -linguistic -vince -operative -brewery -##ont -unemployment -boyd -##ured -##outs -albeit -prophet -1813 -bi -##rr -##face -##rad -quarterly -asteroid -cleaned -radius -temper -##llen -telugu -jerk -viscount -menu -##ote -glimpse -##aya -yacht -hawaiian -baden -##rl -laptop -readily -##gu -monetary -offshore -scots -watches -##yang -##arian -upgrade -needle -xbox -lea -encyclopedia -flank -fingertips -##pus -delight -teachings -confirm -roth -beaches -midway -winters -##iah -teasing -daytime -beverly -gambling -bonnie -##backs -regulated -clement -hermann -tricks -knot -##shing -##uring -##vre -detached -ecological -owed -specialty -byron -inventor -bats -stays -screened -unesco -midland -trim -affection -##ander -##rry -jess -thoroughly -feedback -##uma -chennai -strained -heartbeat -wrapping -overtime -pleaded -##sworth -mon -leisure -oclc -##tate -##ele -feathers -angelo -thirds -nuts -surveys -clever -gill -commentator -##dos -darren -rides -gibraltar -##nc -##mu -dissolution -dedication -shin -meals -saddle -elvis -reds -chaired -taller -appreciation -functioning -niece -favored -advocacy -robbie -criminals -suffolk -yugoslav -passport -constable -congressman -hastings -vera -##rov -consecrated -sparks -ecclesiastical -confined -##ovich -muller -floyd -nora -1822 -paved -1827 -cumberland -ned -saga -spiral -##flow -appreciated -yi -collaborative -treating -similarities -feminine -finishes -##ib -jade -import -##nse -##hot -champagne -mice -securing -celebrities -helsinki -attributes -##gos -cousins -phases -ache -lucia -gandhi -submission -vicar -spear -shine -tasmania -biting -detention -constitute -tighter -seasonal -##gus -terrestrial -matthews -##oka -effectiveness -parody -philharmonic -##onic -1816 -strangers -encoded -consortium -guaranteed -regards -shifts -tortured -collision -supervisor -inform -broader -insight -theaters -armour -emeritus -blink -incorporates -mapping -##50 -##ein -handball -flexible -##nta -substantially -generous -thief -##own -carr -loses -1793 -prose -ucla -romeo -generic -metallic -realization -damages -mk -commissioners -zach -default -##ther -helicopters -lengthy -stems -spa -partnered -spectators -rogue -indication -penalties -teresa -1801 -sen -##tric -dalton -##wich -irving -photographic -##vey -dell -deaf -peters -excluded -unsure -##vable -patterson -crawled -##zio -resided -whipped -latvia -slower -ecole -pipes -employers -maharashtra -comparable -va -textile -pageant -##gel -alphabet -binary -irrigation -chartered -choked -antoine -offs -waking -supplement -##wen -quantities -demolition -regain -locate -urdu -folks -alt -114 -##mc -scary -andreas -whites -##ava -classrooms -mw -aesthetic -publishes -valleys -guides -cubs -johannes -bryant -conventions -affecting -##itt -drain -awesome -isolation -prosecutor -ambitious -apology -captive -downs -atmospheric -lorenzo -aisle -beef -foul -##onia -kidding -composite -disturbed -illusion -natives -##ffer -emi -rockets -riverside -wartime -painters -adolf -melted -##ail -uncertainty -simulation -hawks -progressed -meantime -builder -spray -breach -unhappy -regina -russians -##urg -determining -##tation -tram -1806 -##quin -aging -##12 -1823 -garion -rented -mister -diaz -terminated -clip -1817 -depend -nervously -disco -owe -defenders -shiva -notorious -disbelief -shiny -worcester -##gation -##yr -trailing -undertook -islander -belarus -limitations -watershed -fuller -overlooking -utilized -raphael -1819 -synthetic -breakdown -klein -##nate -moaned -memoir -lamb -practicing -##erly -cellular -arrows -exotic -##graphy -witches -117 -charted -rey -hut -hierarchy -subdivision -freshwater -giuseppe -aloud -reyes -qatar -marty -sideways -utterly -sexually -jude -prayers -mccarthy -softball -blend -damien -##gging -##metric -wholly -erupted -lebanese -negro -revenues -tasted -comparative -teamed -transaction -labeled -maori -sovereignty -parkway -trauma -gran -malay -121 -advancement -descendant -2020 -buzz -salvation -inventory -symbolic -##making -antarctica -mps -##gas -##bro -mohammed -myanmar -holt -submarines -tones -##lman -locker -patriarch -bangkok -emerson -remarks -predators -kin -afghan -confession -norwich -rental -emerge -advantages -##zel -rca -##hold -shortened -storms -aidan -##matic -autonomy -compliance -##quet -dudley -atp -##osis -1803 -motto -documentation -summary -professors -spectacular -christina -archdiocese -flashing -innocence -remake -##dell -psychic -reef -scare -employ -rs -sticks -meg -gus -leans -##ude -accompany -bergen -tomas -##iko -doom -wages -pools -##nch -##bes -breasts -scholarly -alison -outline -brittany -breakthrough -willis -realistic -##cut -##boro -competitor -##stan -pike -picnic -icon -designing -commercials -washing -villain -skiing -micro -costumes -auburn -halted -executives -##hat -logistics -cycles -vowel -applicable -barrett -exclaimed -eurovision -eternity -ramon -##umi -##lls -modifications -sweeping -disgust -##uck -torch -aviv -ensuring -rude -dusty -sonic -donovan -outskirts -cu -pathway -##band -##gun -##lines -disciplines -acids -cadet -paired -##40 -sketches -##sive -marriages -##⁺ -folding -peers -slovak -implies -admired -##beck -1880s -leopold -instinct -attained -weston -megan -horace -##ination -dorsal -ingredients -evolutionary -##its -complications -deity -lethal -brushing -levy -deserted -institutes -posthumously -delivering -telescope -coronation -motivated -rapids -luc -flicked -pays -volcano -tanner -weighed -##nica -crowds -frankie -gifted -addressing -granddaughter -winding -##rna -constantine -gomez -##front -landscapes -rudolf -anthropology -slate -werewolf -##lio -astronomy -circa -rouge -dreaming -sack -knelt -drowned -naomi -prolific -tracked -freezing -herb -##dium -agony -randall -twisting -wendy -deposit -touches -vein -wheeler -##bbled -##bor -batted -retaining -tire -presently -compare -specification -daemon -nigel -##grave -merry -recommendation -czechoslovakia -sandra -ng -roma -##sts -lambert -inheritance -sheikh -winchester -cries -examining -##yle -comeback -cuisine -nave -##iv -ko -retrieve -tomatoes -barker -polished -defining -irene -lantern -personalities -begging -tract -swore -1809 -175 -##gic -omaha -brotherhood -##rley -haiti -##ots -exeter -##ete -##zia -steele -dumb -pearson -210 -surveyed -elisabeth -trends -##ef -fritz -##rf -premium -bugs -fraction -calmly -viking -##birds -tug -inserted -unusually -##ield -confronted -distress -crashing -brent -turks -resign -##olo -cambodia -gabe -sauce -##kal -evelyn -116 -extant -clusters -quarry -teenagers -luna -##lers -##ister -affiliation -drill -##ashi -panthers -scenic -libya -anita -strengthen -inscriptions -##cated -lace -sued -judith -riots -##uted -mint -##eta -preparations -midst -dub -challenger -##vich -mock -cf -displaced -wicket -breaths -enables -schmidt -analyst -##lum -ag -highlight -automotive -axe -josef -newark -sufficiently -resembles -50th -##pal -flushed -mum -traits -##ante -commodore -incomplete -warming -titular -ceremonial -ethical -118 -celebrating -eighteenth -cao -lima -medalist -mobility -strips -snakes -##city -miniature -zagreb -barton -escapes -umbrella -automated -doubted -differs -cooled -georgetown -dresden -cooked -fade -wyatt -rna -jacobs -carlton -abundant -stereo -boost -madras -inning -##hia -spur -ip -malayalam -begged -osaka -groan -escaping -charging -dose -vista -##aj -bud -papa -communists -advocates -edged -tri -##cent -resemble -peaking -necklace -fried -montenegro -saxony -goose -glances -stuttgart -curator -recruit -grocery -sympathetic -##tting -##fort -127 -lotus -randolph -ancestor -##rand -succeeding -jupiter -1798 -macedonian -##heads -hiking -1808 -handing -fischer -##itive -garbage -node -##pies -prone -singular -papua -inclined -attractions -italia -pouring -motioned -grandma -garnered -jacksonville -corp -ego -ringing -aluminum -##hausen -ordering -##foot -drawer -traders -synagogue -##play -##kawa -resistant -wandering -fragile -fiona -teased -var -hardcore -soaked -jubilee -decisive -exposition -mercer -poster -valencia -hale -kuwait -1811 -##ises -##wr -##eed -tavern -gamma -122 -johan -##uer -airways -amino -gil -##ury -vocational -domains -torres -##sp -generator -folklore -outcomes -##keeper -canberra -shooter -fl -beams -confrontation -##lling -##gram -feb -aligned -forestry -pipeline -jax -motorway -conception -decay -##tos -coffin -##cott -stalin -1805 -escorted -minded -##nam -sitcom -purchasing -twilight -veronica -additions -passive -tensions -straw -123 -frequencies -1804 -refugee -cultivation -##iate -christie -clary -bulletin -crept -disposal -##rich -##zong -processor -crescent -##rol -bmw -emphasized -whale -nazis -aurora -##eng -dwelling -hauled -sponsors -toledo -mega -ideology -theatres -tessa -cerambycidae -saves -turtle -cone -suspects -kara -rusty -yelling -greeks -mozart -shades -cocked -participant -##tro -shire -spit -freeze -necessity -##cos -inmates -nielsen -councillors -loaned -uncommon -omar -peasants -botanical -offspring -daniels -formations -jokes -1794 -pioneers -sigma -licensing -##sus -wheelchair -polite -1807 -liquor -pratt -trustee -##uta -forewings -balloon -##zz -kilometre -camping -explicit -casually -shawn -foolish -teammates -nm -hassan -carrie -judged -satisfy -vanessa -knives -selective -cnn -flowed -##lice -eclipse -stressed -eliza -mathematician -cease -cultivated -##roy -commissions -browns -##ania -destroyers -sheridan -meadow -##rius -minerals -##cial -downstream -clash -gram -memoirs -ventures -baha -seymour -archie -midlands -edith -fare -flynn -invite -canceled -tiles -stabbed -boulder -incorporate -amended -camden -facial -mollusk -unreleased -descriptions -yoga -grabs -550 -raises -ramp -shiver -##rose -coined -pioneering -tunes -qing -warwick -tops -119 -melanie -giles -##rous -wandered -##inal -annexed -nov -30th -unnamed -##ished -organizational -airplane -normandy -stoke -whistle -blessing -violations -chased -holders -shotgun -##ctic -outlet -reactor -##vik -tires -tearing -shores -fortified -mascot -constituencies -nc -columnist -productive -tibet -##rta -lineage -hooked -oct -tapes -judging -cody -##gger -hansen -kashmir -triggered -##eva -solved -cliffs -##tree -resisted -anatomy -protesters -transparent -implied -##iga -injection -mattress -excluding -##mbo -defenses -helpless -devotion -##elli -growl -liberals -weber -phenomena -atoms -plug -##iff -mortality -apprentice -howe -convincing -aaa -swimmer -barber -leone -promptly -sodium -def -nowadays -arise -##oning -gloucester -corrected -dignity -norm -erie -##ders -elders -evacuated -sylvia -compression -##yar -hartford -pose -backpack -reasoning -accepts -24th -wipe -millimetres -marcel -##oda -dodgers -albion -1790 -overwhelmed -aerospace -oaks -1795 -showcase -acknowledge -recovering -nolan -ashe -hurts -geology -fashioned -disappearance -farewell -swollen -shrug -marquis -wimbledon -124 -rue -1792 -commemorate -reduces -experiencing -inevitable -calcutta -intel -##court -murderer -sticking -fisheries -imagery -bloom -280 -brake -##inus -gustav -hesitation -memorable -po -viral -beans -accidents -tunisia -antenna -spilled -consort -treatments -aye -perimeter -##gard -donation -hostage -migrated -banker -addiction -apex -lil -trout -##ously -conscience -##nova -rams -sands -genome -passionate -troubles -##lets -##set -amid -##ibility -##ret -higgins -exceed -vikings -##vie -payne -##zan -muscular -##ste -defendant -sucking -##wal -ibrahim -fuselage -claudia -vfl -europeans -snails -interval -##garh -preparatory -statewide -tasked -lacrosse -viktor -##lation -angola -##hra -flint -implications -employs -teens -patrons -stall -weekends -barriers -scrambled -nucleus -tehran -jenna -parsons -lifelong -robots -displacement -5000 -##bles -precipitation -##gt -knuckles -clutched -1802 -marrying -ecology -marx -accusations -declare -scars -kolkata -mat -meadows -bermuda -skeleton -finalists -vintage -crawl -coordinate -affects -subjected -orchestral -mistaken -##tc -mirrors -dipped -relied -260 -arches -candle -##nick -incorporating -wildly -fond -basilica -owl -fringe -rituals -whispering -stirred -feud -tertiary -slick -goat -honorable -whereby -skip -ricardo -stripes -parachute -adjoining -submerged -synthesizer -##gren -intend -positively -ninety -phi -beaver -partition -fellows -alexis -prohibition -carlisle -bizarre -fraternity -##bre -doubts -icy -cbc -aquatic -sneak -sonny -combines -airports -crude -supervised -spatial -merge -alfonso -##bic -corrupt -scan -undergo -##ams -disabilities -colombian -comparing -dolphins -perkins -##lish -reprinted -unanimous -bounced -hairs -underworld -midwest -semester -bucket -paperback -miniseries -coventry -demise -##leigh -demonstrations -sensor -rotating -yan -##hler -arrange -soils -##idge -hyderabad -labs -##dr -brakes -grandchildren -##nde -negotiated -rover -ferrari -continuation -directorate -augusta -stevenson -counterpart -gore -##rda -nursery -rican -ave -collectively -broadly -pastoral -repertoire -asserted -discovering -nordic -styled -fiba -cunningham -harley -middlesex -survives -tumor -tempo -zack -aiming -lok -urgent -##rade -##nto -devils -##ement -contractor -turin -##wl -##ool -bliss -repaired -simmons -moan -astronomical -cr -negotiate -lyric -1890s -lara -bred -clad -angus -pbs -##ience -engineered -posed -##lk -hernandez -possessions -elbows -psychiatric -strokes -confluence -electorate -lifts -campuses -lava -alps -##ep -##ution -##date -physicist -woody -##page -##ographic -##itis -juliet -reformation -sparhawk -320 -complement -suppressed -jewel -##½ -floated -##kas -continuity -sadly -##ische -inability -melting -scanning -paula -flour -judaism -safer -vague -##lm -solving -curb -##stown -financially -gable -bees -expired -miserable -cassidy -dominion -1789 -cupped -145 -robbery -facto -amos -warden -resume -tallest -marvin -ing -pounded -usd -declaring -gasoline -##aux -darkened -270 -650 -sophomore -##mere -erection -gossip -televised -risen -dial -##eu -pillars -##link -passages -profound -##tina -arabian -ashton -silicon -nail -##ead -##lated -##wer -##hardt -fleming -firearms -ducked -circuits -blows -waterloo -titans -##lina -atom -fireplace -cheshire -financed -activation -algorithms -##zzi -constituent -catcher -cherokee -partnerships -sexuality -platoon -tragic -vivian -guarded -whiskey -meditation -poetic -##late -##nga -##ake -porto -listeners -dominance -kendra -mona -chandler -factions -22nd -salisbury -attitudes -derivative -##ido -##haus -intake -paced -javier -illustrator -barrels -bias -cockpit -burnett -dreamed -ensuing -##anda -receptors -someday -hawkins -mattered -##lal -slavic -1799 -jesuit -cameroon -wasted -tai -wax -lowering -victorious -freaking -outright -hancock -librarian -sensing -bald -calcium -myers -tablet -announcing -barack -shipyard -pharmaceutical -##uan -greenwich -flush -medley -patches -wolfgang -pt -speeches -acquiring -exams -nikolai -##gg -hayden -kannada -##type -reilly -##pt -waitress -abdomen -devastated -capped -pseudonym -pharmacy -fulfill -paraguay -1796 -clicked -##trom -archipelago -syndicated -##hman -lumber -orgasm -rejection -clifford -lorraine -advent -mafia -rodney -brock -##ght -##used -##elia -cassette -chamberlain -despair -mongolia -sensors -developmental -upstream -##eg -##alis -spanning -165 -trombone -basque -seeded -interred -renewable -rhys -leapt -revision -molecule -##ages -chord -vicious -nord -shivered -23rd -arlington -debts -corpus -sunrise -bays -blackburn -centimetres -##uded -shuddered -gm -strangely -gripping -cartoons -isabelle -orbital -##ppa -seals -proving -##lton -refusal -strengthened -bust -assisting -baghdad -batsman -portrayal -mara -pushes -spears -og -##cock -reside -nathaniel -brennan -1776 -confirmation -caucus -##worthy -markings -yemen -nobles -ku -lazy -viewer -catalan -encompasses -sawyer -##fall -sparked -substances -patents -braves -arranger -evacuation -sergio -persuade -dover -tolerance -penguin -cum -jockey -insufficient -townships -occupying -declining -plural -processed -projection -puppet -flanders -introduces -liability -##yon -gymnastics -antwerp -taipei -hobart -candles -jeep -wes -observers -126 -chaplain -bundle -glorious -##hine -hazel -flung -sol -excavations -dumped -stares -sh -bangalore -triangular -icelandic -intervals -expressing -turbine -##vers -songwriting -crafts -##igo -jasmine -ditch -rite -##ways -entertaining -comply -sorrow -wrestlers -basel -emirates -marian -rivera -helpful -##some -caution -downward -networking -##atory -##tered -darted -genocide -emergence -replies -specializing -spokesman -convenient -unlocked -fading -augustine -concentrations -resemblance -elijah -investigator -andhra -##uda -promotes -bean -##rrell -fleeing -wan -simone -announcer -##ame -##bby -lydia -weaver -132 -residency -modification -##fest -stretches -##ast -alternatively -nat -lowe -lacks -##ented -pam -tile -concealed -inferior -abdullah -residences -tissues -vengeance -##ided -moisture -peculiar -groove -zip -bologna -jennings -ninja -oversaw -zombies -pumping -batch -livingston -emerald -installations -1797 -peel -nitrogen -rama -##fying -##star -schooling -strands -responding -werner -##ost -lime -casa -accurately -targeting -##rod -underway -##uru -hemisphere -lester -##yard -occupies -2d -griffith -angrily -reorganized -##owing -courtney -deposited -##dd -##30 -estadio -##ifies -dunn -exiled -##ying -checks -##combe -##о -##fly -successes -unexpectedly -blu -assessed -##flower -##ه -observing -sacked -spiders -kn -##tail -mu -nodes -prosperity -audrey -divisional -155 -broncos -tangled -adjust -feeds -erosion -paolo -surf -directory -snatched -humid -admiralty -screwed -gt -reddish -##nese -modules -trench -lamps -bind -leah -bucks -competes -##nz -##form -transcription -##uc -isles -violently -clutching -pga -cyclist -inflation -flats -ragged -unnecessary -##hian -stubborn -coordinated -harriet -baba -disqualified -330 -insect -wolfe -##fies -reinforcements -rocked -duel -winked -embraced -bricks -##raj -hiatus -defeats -pending -brightly -jealousy -##xton -##hm -##uki -lena -gdp -colorful -##dley -stein -kidney -##shu -underwear -wanderers -##haw -##icus -guardians -m³ -roared -habits -##wise -permits -gp -uranium -punished -disguise -bundesliga -elise -dundee -erotic -partisan -pi -collectors -float -individually -rendering -behavioral -bucharest -ser -hare -valerie -corporal -nutrition -proportional -##isa -immense -##kis -pavement -##zie -##eld -sutherland -crouched -1775 -##lp -suzuki -trades -endurance -operas -crosby -prayed -priory -rory -socially -##urn -gujarat -##pu -walton -cube -pasha -privilege -lennon -floods -thorne -waterfall -nipple -scouting -approve -##lov -minorities -voter -dwight -extensions -assure -ballroom -slap -dripping -privileges -rejoined -confessed -demonstrating -patriotic -yell -investor -##uth -pagan -slumped -squares -##cle -##kins -confront -bert -embarrassment -##aid -aston -urging -sweater -starr -yuri -brains -williamson -commuter -mortar -structured -selfish -exports -##jon -cds -##him -unfinished -##rre -mortgage -destinations -##nagar -canoe -solitary -buchanan -delays -magistrate -fk -##pling -motivation -##lier -##vier -recruiting -assess -##mouth -malik -antique -1791 -pius -rahman -reich -tub -zhou -smashed -airs -galway -xii -conditioning -honduras -discharged -dexter -##pf -lionel -129 -debates -lemon -tiffany -volunteered -dom -dioxide -procession -devi -sic -tremendous -advertisements -colts -transferring -verdict -hanover -decommissioned -utter -relate -pac -racism -##top -beacon -limp -similarity -terra -occurrence -ant -##how -becky -capt -updates -armament -richie -pal -##graph -halloween -mayo -##ssen -##bone -cara -serena -fcc -dolls -obligations -##dling -violated -lafayette -jakarta -exploitation -##ime -infamous -iconic -##lah -##park -kitty -moody -reginald -dread -spill -crystals -olivier -modeled -bluff -equilibrium -separating -notices -ordnance -extinction -onset -cosmic -attachment -sammy -expose -privy -anchored -##bil -abbott -admits -bending -baritone -emmanuel -policeman -vaughan -winged -climax -dresses -denny -polytechnic -mohamed -burmese -authentic -nikki -genetics -grandparents -homestead -gaza -postponed -metacritic -una -##sby -##bat -unstable -dissertation -##rial -##cian -curls -obscure -uncovered -bronx -praying -disappearing -##hoe -prehistoric -coke -turret -mutations -nonprofit -pits -monaco -##ي -##usion -prominently -dispatched -podium -##mir -uci -##uation -133 -fortifications -birthplace -kendall -##lby -##oll -preacher -rack -goodman -##rman -persistent -##ott -countless -jaime -recorder -lexington -persecution -jumps -renewal -wagons -##11 -crushing -##holder -decorations -##lake -abundance -wrath -laundry -£1 -garde -##rp -jeanne -beetles -peasant -##sl -splitting -caste -sergei -##rer -##ema -scripts -##ively -rub -satellites -##vor -inscribed -verlag -scrapped -gale -packages -chick -potato -slogan -kathleen -arabs -##culture -counterparts -reminiscent -choral -##tead -rand -retains -bushes -dane -accomplish -courtesy -closes -##oth -slaughter -hague -krakow -lawson -tailed -elias -ginger -##ttes -canopy -betrayal -rebuilding -turf -##hof -frowning -allegiance -brigades -kicks -rebuild -polls -alias -nationalism -td -rowan -audition -bowie -fortunately -recognizes -harp -dillon -horrified -##oro -renault -##tics -ropes -##α -presumed -rewarded -infrared -wiping -accelerated -illustration -##rid -presses -practitioners -badminton -##iard -detained -##tera -recognizing -relates -misery -##sies -##tly -reproduction -piercing -potatoes -thornton -esther -manners -hbo -##aan -ours -bullshit -ernie -perennial -sensitivity -illuminated -rupert -##jin -##iss -##ear -rfc -nassau -##dock -staggered -socialism -##haven -appointments -nonsense -prestige -sharma -haul -##tical -solidarity -gps -##ook -##rata -igor -pedestrian -##uit -baxter -tenants -wires -medication -unlimited -guiding -impacts -diabetes -##rama -sasha -pas -clive -extraction -131 -continually -constraints -##bilities -sonata -hunted -sixteenth -chu -planting -quote -mayer -pretended -abs -spat -##hua -ceramic -##cci -curtains -pigs -pitching -##dad -latvian -sore -dayton -##sted -##qi -patrols -slice -playground -##nted -shone -stool -apparatus -inadequate -mates -treason -##ija -desires -##liga -##croft -somalia -laurent -mir -leonardo -oracle -grape -obliged -chevrolet -thirteenth -stunning -enthusiastic -##ede -accounted -concludes -currents -basil -##kovic -drought -##rica -mai -##aire -shove -posting -##shed -pilgrimage -humorous -packing -fry -pencil -wines -smells -144 -marilyn -aching -newest -clung -bon -neighbours -sanctioned -##pie -mug -##stock -drowning -##mma -hydraulic -##vil -hiring -reminder -lilly -investigators -##ncies -sour -##eous -compulsory -packet -##rion -##graphic -##elle -cannes -##inate -depressed -##rit -heroic -importantly -theresa -##tled -conway -saturn -marginal -rae -##xia -corresponds -royce -pact -jasper -explosives -packaging -aluminium -##ttered -denotes -rhythmic -spans -assignments -hereditary -outlined -originating -sundays -lad -reissued -greeting -beatrice -##dic -pillar -marcos -plots -handbook -alcoholic -judiciary -avant -slides -extract -masculine -blur -##eum -##force -homage -trembled -owens -hymn -trey -omega -signaling -socks -accumulated -reacted -attic -theo -lining -angie -distraction -primera -talbot -##key -1200 -ti -creativity -billed -##hey -deacon -eduardo -identifies -proposition -dizzy -gunner -hogan -##yam -##pping -##hol -ja -##chan -jensen -reconstructed -##berger -clearance -darius -##nier -abe -harlem -plea -dei -circled -emotionally -notation -fascist -neville -exceeded -upwards -viable -ducks -##fo -workforce -racer -limiting -shri -##lson -possesses -1600 -kerr -moths -devastating -laden -disturbing -locking -##cture -gal -fearing -accreditation -flavor -aide -1870s -mountainous -##baum -melt -##ures -motel -texture -servers -soda -##mb -herd -##nium -erect -puzzled -hum -peggy -examinations -gould -testified -geoff -ren -devised -sacks -##law -denial -posters -grunted -cesar -tutor -ec -gerry -offerings -byrne -falcons -combinations -ct -incoming -pardon -rocking -26th -avengers -flared -mankind -seller -uttar -loch -nadia -stroking -exposing -##hd -fertile -ancestral -instituted -##has -noises -prophecy -taxation -eminent -vivid -pol -##bol -dart -indirect -multimedia -notebook -upside -displaying -adrenaline -referenced -geometric -##iving -progression -##ddy -blunt -announce -##far -implementing -##lav -aggression -liaison -cooler -cares -headache -plantations -gorge -dots -impulse -thickness -ashamed -averaging -kathy -obligation -precursor -137 -fowler -symmetry -thee -225 -hears -##rai -undergoing -ads -butcher -bowler -##lip -cigarettes -subscription -goodness -##ically -browne -##hos -##tech -kyoto -donor -##erty -damaging -friction -drifting -expeditions -hardened -prostitution -152 -fauna -blankets -claw -tossing -snarled -butterflies -recruits -investigative -coated -healed -138 -communal -hai -xiii -academics -boone -psychologist -restless -lahore -stephens -mba -brendan -foreigners -printer -##pc -ached -explode -27th -deed -scratched -dared -##pole -cardiac -1780 -okinawa -proto -commando -compelled -oddly -electrons -##base -replica -thanksgiving -##rist -sheila -deliberate -stafford -tidal -representations -hercules -ou -##path -##iated -kidnapping -lenses -##tling -deficit -samoa -mouths -consuming -computational -maze -granting -smirk -razor -fixture -ideals -inviting -aiden -nominal -##vs -issuing -julio -pitt -ramsey -docks -##oss -exhaust -##owed -bavarian -draped -anterior -mating -ethiopian -explores -noticing -##nton -discarded -convenience -hoffman -endowment -beasts -cartridge -mormon -paternal -probe -sleeves -interfere -lump -deadline -##rail -jenks -bulldogs -scrap -alternating -justified -reproductive -nam -seize -descending -secretariat -kirby -coupe -grouped -smash -panther -sedan -tapping -##18 -lola -cheer -germanic -unfortunate -##eter -unrelated -##fan -subordinate -##sdale -suzanne -advertisement -##ility -horsepower -##lda -cautiously -discourse -luigi -##mans -##fields -noun -prevalent -mao -schneider -everett -surround -governorate -kira -##avia -westward -##take -misty -rails -sustainability -134 -unused -##rating -packs -toast -unwilling -regulate -thy -suffrage -nile -awe -assam -definitions -travelers -affordable -##rb -conferred -sells -undefeated -beneficial -torso -basal -repeating -remixes -##pass -bahrain -cables -fang -##itated -excavated -numbering -statutory -##rey -deluxe -##lian -forested -ramirez -derbyshire -zeus -slamming -transfers -astronomer -banana -lottery -berg -histories -bamboo -##uchi -resurrection -posterior -bowls -vaguely -##thi -thou -preserving -tensed -offence -##inas -meyrick -callum -ridden -watt -langdon -tying -lowland -snorted -daring -truman -##hale -##girl -aura -overly -filing -weighing -goa -infections -philanthropist -saunders -eponymous -##owski -latitude -perspectives -reviewing -mets -commandant -radial -##kha -flashlight -reliability -koch -vowels -amazed -ada -elaine -supper -##rth -##encies -predator -debated -soviets -cola -##boards -##nah -compartment -crooked -arbitrary -fourteenth -##ctive -havana -majors -steelers -clips -profitable -ambush -exited -packers -##tile -nude -cracks -fungi -##е -limb -trousers -josie -shelby -tens -frederic -##ος -definite -smoothly -constellation -insult -baton -discs -lingering -##nco -conclusions -lent -staging -becker -grandpa -shaky -##tron -einstein -obstacles -sk -adverse -elle -economically -##moto -mccartney -thor -dismissal -motions -readings -nostrils -treatise -##pace -squeezing -evidently -prolonged -1783 -venezuelan -je -marguerite -beirut -takeover -shareholders -##vent -denise -digit -airplay -norse -##bbling -imaginary -pills -hubert -blaze -vacated -eliminating -##ello -vine -mansfield -##tty -retrospective -barrow -borne -clutch -bail -forensic -weaving -##nett -##witz -desktop -citadel -promotions -worrying -dorset -ieee -subdivided -##iating -manned -expeditionary -pickup -synod -chuckle -185 -barney -##rz -##ffin -functionality -karachi -litigation -meanings -uc -lick -turbo -anders -##ffed -execute -curl -oppose -ankles -typhoon -##د -##ache -##asia -linguistics -compassion -pressures -grazing -perfection -##iting -immunity -monopoly -muddy -backgrounds -136 -namibia -francesca -monitors -attracting -stunt -tuition -##ии -vegetable -##mates -##quent -mgm -jen -complexes -forts -##ond -cellar -bites -seventeenth -royals -flemish -failures -mast -charities -##cular -peruvian -capitals -macmillan -ipswich -outward -frigate -postgraduate -folds -employing -##ouse -concurrently -fiery -##tai -contingent -nightmares -monumental -nicaragua -##kowski -lizard -mal -fielding -gig -reject -##pad -harding -##ipe -coastline -##cin -##nos -beethoven -humphrey -innovations -##tam -##nge -norris -doris -solicitor -huang -obey -141 -##lc -niagara -##tton -shelves -aug -bourbon -curry -nightclub -specifications -hilton -##ndo -centennial -dispersed -worm -neglected -briggs -sm -font -kuala -uneasy -plc -##nstein -##bound -##aking -##burgh -awaiting -pronunciation -##bbed -##quest -eh -optimal -zhu -raped -greens -presided -brenda -worries -##life -venetian -marxist -turnout -##lius -refined -braced -sins -grasped -sunderland -nickel -speculated -lowell -cyrillic -communism -fundraising -resembling -colonists -mutant -freddie -usc -##mos -gratitude -##run -mural -##lous -chemist -wi -reminds -28th -steals -tess -pietro -##ingen -promoter -ri -microphone -honoured -rai -sant -##qui -feather -##nson -burlington -kurdish -terrorists -deborah -sickness -##wed -##eet -hazard -irritated -desperation -veil -clarity -##rik -jewels -xv -##gged -##ows -##cup -berkshire -unfair -mysteries -orchid -winced -exhaustion -renovations -stranded -obe -infinity -##nies -adapt -redevelopment -thanked -registry -olga -domingo -noir -tudor -ole -##atus -commenting -behaviors -##ais -crisp -pauline -probable -stirling -wigan -##bian -paralympics -panting -surpassed -##rew -luca -barred -pony -famed -##sters -cassandra -waiter -carolyn -exported -##orted -andres -destructive -deeds -jonah -castles -vacancy -suv -##glass -1788 -orchard -yep -famine -belarusian -sprang -##forth -skinny -##mis -administrators -rotterdam -zambia -zhao -boiler -discoveries -##ride -##physics -lucius -disappointing -outreach -spoon -##frame -qualifications -unanimously -enjoys -regency -##iidae -stade -realism -veterinary -rodgers -dump -alain -chestnut -castile -censorship -rumble -gibbs -##itor -communion -reggae -inactivated -logs -loads -##houses -homosexual -##iano -ale -informs -##cas -phrases -plaster -linebacker -ambrose -kaiser -fascinated -850 -limerick -recruitment -forge -mastered -##nding -leinster -rooted -threaten -##strom -borneo -##hes -suggestions -scholarships -propeller -documentaries -patronage -coats -constructing -invest -neurons -comet -entirety -shouts -identities -annoying -unchanged -wary -##antly -##ogy -neat -oversight -##kos -phillies -replay -constance -##kka -incarnation -humble -skies -minus -##acy -smithsonian -##chel -guerrilla -jar -cadets -##plate -surplus -audit -##aru -cracking -joanna -louisa -pacing -##lights -intentionally -##iri -diner -nwa -imprint -australians -tong -unprecedented -bunker -naive -specialists -ark -nichols -railing -leaked -pedal -##uka -shrub -longing -roofs -v8 -captains -neural -tuned -##ntal -##jet -emission -medina -frantic -codex -definitive -sid -abolition -intensified -stocks -enrique -sustain -genoa -oxide -##written -clues -cha -##gers -tributaries -fragment -venom -##rity -##ente -##sca -muffled -vain -sire -laos -##ingly -##hana -hastily -snapping -surfaced -sentiment -motive -##oft -contests -approximate -mesa -luckily -dinosaur -exchanges -propelled -accord -bourne -relieve -tow -masks -offended -##ues -cynthia -##mmer -rains -bartender -zinc -reviewers -lois -##sai -legged -arrogant -rafe -rosie -comprise -handicap -blockade -inlet -lagoon -copied -drilling -shelley -petals -##inian -mandarin -obsolete -##inated -onward -arguably -productivity -cindy -praising -seldom -busch -discusses -raleigh -shortage -ranged -stanton -encouragement -firstly -conceded -overs -temporal -##uke -cbe -##bos -woo -certainty -pumps -##pton -stalked -##uli -lizzie -periodic -thieves -weaker -##night -gases -shoving -chooses -wc -##chemical -prompting -weights -##kill -robust -flanked -sticky -hu -tuberculosis -##eb -##eal -christchurch -resembled -wallet -reese -inappropriate -pictured -distract -fixing -fiddle -giggled -burger -heirs -hairy -mechanic -torque -apache -obsessed -chiefly -cheng -logging -##tag -extracted -meaningful -numb -##vsky -gloucestershire -reminding -##bay -unite -##lit -breeds -diminished -clown -glove -1860s -##ن -##ug -archibald -focal -freelance -sliced -depiction -##yk -organism -switches -sights -stray -crawling -##ril -lever -leningrad -interpretations -loops -anytime -reel -alicia -delighted -##ech -inhaled -xiv -suitcase -bernie -vega -licenses -northampton -exclusion -induction -monasteries -racecourse -homosexuality -##right -##sfield -##rky -dimitri -michele -alternatives -ions -commentators -genuinely -objected -pork -hospitality -fencing -stephan -warships -peripheral -wit -drunken -wrinkled -quentin -spends -departing -chung -numerical -spokesperson -##zone -johannesburg -caliber -killers -##udge -assumes -neatly -demographic -abigail -bloc -##vel -mounting -##lain -bentley -slightest -xu -recipients -##jk -merlin -##writer -seniors -prisons -blinking -hindwings -flickered -kappa -##hel -80s -strengthening -appealing -brewing -gypsy -mali -lashes -hulk -unpleasant -harassment -bio -treaties -predict -instrumentation -pulp -troupe -boiling -mantle -##ffe -ins -##vn -dividing -handles -verbs -##onal -coconut -senegal -340 -thorough -gum -momentarily -##sto -cocaine -panicked -destined -##turing -teatro -denying -weary -captained -mans -##hawks -##code -wakefield -bollywood -thankfully -##16 -cyril -##wu -amendments -##bahn -consultation -stud -reflections -kindness -1787 -internally -##ovo -tex -mosaic -distribute -paddy -seeming -143 -##hic -piers -##15 -##mura -##verse -popularly -winger -kang -sentinel -mccoy -##anza -covenant -##bag -verge -fireworks -suppress -thrilled -dominate -##jar -swansea -##60 -142 -reconciliation -##ndi -stiffened -cue -dorian -##uf -damascus -amor -ida -foremost -##aga -porsche -unseen -dir -##had -##azi -stony -lexi -melodies -##nko -angular -integer -podcast -ants -inherent -jaws -justify -persona -##olved -josephine -##nr -##ressed -customary -flashes -gala -cyrus -glaring -backyard -ariel -physiology -greenland -html -stir -avon -atletico -finch -methodology -ked -##lent -mas -catholicism -townsend -branding -quincy -fits -containers -1777 -ashore -aragon -##19 -forearm -poisoning -##sd -adopting -conquer -grinding -amnesty -keller -finances -evaluate -forged -lankan -instincts -##uto -guam -bosnian -photographed -workplace -desirable -protector -##dog -allocation -intently -encourages -willy -##sten -bodyguard -electro -brighter -##ν -bihar -##chev -lasts -opener -amphibious -sal -verde -arte -##cope -captivity -vocabulary -yields -##tted -agreeing -desmond -pioneered -##chus -strap -campaigned -railroads -##ович -emblem -##dre -stormed -501 -##ulous -marijuana -northumberland -##gn -##nath -bowen -landmarks -beaumont -##qua -danube -##bler -attorneys -th -ge -flyers -critique -villains -cass -mutation -acc -##0s -colombo -mckay -motif -sampling -concluding -syndicate -##rell -neon -stables -ds -warnings -clint -mourning -wilkinson -##tated -merrill -leopard -evenings -exhaled -emil -sonia -ezra -discrete -stove -farrell -fifteenth -prescribed -superhero -##rier -worms -helm -wren -##duction -##hc -expo -##rator -hq -unfamiliar -antony -prevents -acceleration -fiercely -mari -painfully -calculations -cheaper -ign -clifton -irvine -davenport -mozambique -##np -pierced -##evich -wonders -##wig -##cate -##iling -crusade -ware -##uel -enzymes -reasonably -mls -##coe -mater -ambition -bunny -eliot -kernel -##fin -asphalt -headmaster -torah -aden -lush -pins -waived -##care -##yas -joao -substrate -enforce -##grad -##ules -alvarez -selections -epidemic -tempted -##bit -bremen -translates -ensured -waterfront -29th -forrest -manny -malone -kramer -reigning -cookies -simpler -absorption -205 -engraved -##ffy -evaluated -1778 -haze -146 -comforting -crossover -##abe -thorn -##rift -##imo -##pop -suppression -fatigue -cutter -##tr -201 -wurttemberg -##orf -enforced -hovering -proprietary -gb -samurai -syllable -ascent -lacey -tick -lars -tractor -merchandise -rep -bouncing -defendants -##yre -huntington -##ground -##oko -standardized -##hor -##hima -assassinated -nu -predecessors -rainy -liar -assurance -lyrical -##uga -secondly -flattened -ios -parameter -undercover -##mity -bordeaux -punish -ridges -markers -exodus -inactive -hesitate -debbie -nyc -pledge -savoy -nagar -offset -organist -##tium -hesse -marin -converting -##iver -diagram -propulsion -pu -validity -reverted -supportive -##dc -ministries -clans -responds -proclamation -##inae -##ø -##rea -ein -pleading -patriot -sf -birch -islanders -strauss -hates -##dh -brandenburg -concession -rd -##ob -1900s -killings -textbook -antiquity -cinematography -wharf -embarrassing -setup -creed -farmland -inequality -centred -signatures -fallon -370 -##ingham -##uts -ceylon -gazing -directive -laurie -##tern -globally -##uated -##dent -allah -excavation -threads -##cross -148 -frantically -icc -utilize -determines -respiratory -thoughtful -receptions -##dicate -merging -chandra -seine -147 -builders -builds -diagnostic -dev -visibility -goddamn -analyses -dhaka -cho -proves -chancel -concurrent -curiously -canadians -pumped -restoring -1850s -turtles -jaguar -sinister -spinal -traction -declan -vows -1784 -glowed -capitalism -swirling -install -universidad -##lder -##oat -soloist -##genic -##oor -coincidence -beginnings -nissan -dip -resorts -caucasus -combustion -infectious -##eno -pigeon -serpent -##itating -conclude -masked -salad -jew -##gr -surreal -toni -##wc -harmonica -151 -##gins -##etic -##coat -fishermen -intending -bravery -##wave -klaus -titan -wembley -taiwanese -ransom -40th -incorrect -hussein -eyelids -jp -cooke -dramas -utilities -##etta -##print -eisenhower -principally -granada -lana -##rak -openings -concord -##bl -bethany -connie -morality -sega -##mons -##nard -earnings -##kara -##cine -wii -communes -##rel -coma -composing -softened -severed -grapes -##17 -nguyen -analyzed -warlord -hubbard -heavenly -behave -slovenian -##hit -##ony -hailed -filmmakers -trance -caldwell -skye -unrest -coward -likelihood -##aging -bern -sci -taliban -honolulu -propose -##wang -1700 -browser -imagining -cobra -contributes -dukes -instinctively -conan -violinist -##ores -accessories -gradual -##amp -quotes -sioux -##dating -undertake -intercepted -sparkling -compressed -139 -fungus -tombs -haley -imposing -rests -degradation -lincolnshire -retailers -wetlands -tulsa -distributor -dungeon -nun -greenhouse -convey -atlantis -aft -exits -oman -dresser -lyons -##sti -joking -eddy -judgement -omitted -digits -##cts -##game -juniors -##rae -cents -stricken -une -##ngo -wizards -weir -breton -nan -technician -fibers -liking -royalty -##cca -154 -persia -terribly -magician -##rable -##unt -vance -cafeteria -booker -camille -warmer -##static -consume -cavern -gaps -compass -contemporaries -foyer -soothing -graveyard -maj -plunged -blush -##wear -cascade -demonstrates -ordinance -##nov -boyle -##lana -rockefeller -shaken -banjo -izzy -##ense -breathless -vines -##32 -##eman -alterations -chromosome -dwellings -feudal -mole -153 -catalonia -relics -tenant -mandated -##fm -fridge -hats -honesty -patented -raul -heap -cruisers -accusing -enlightenment -infants -wherein -chatham -contractors -zen -affinity -hc -osborne -piston -156 -traps -maturity -##rana -lagos -##zal -peering -##nay -attendant -dealers -protocols -subset -prospects -biographical -##cre -artery -##zers -insignia -nuns -endured -##eration -recommend -schwartz -serbs -berger -cromwell -crossroads -##ctor -enduring -clasped -grounded -##bine -marseille -twitched -abel -choke -https -catalyst -moldova -italians -##tist -disastrous -wee -##oured -##nti -wwf -nope -##piration -##asa -expresses -thumbs -167 -##nza -coca -1781 -cheating -##ption -skipped -sensory -heidelberg -spies -satan -dangers -semifinal -202 -bohemia -whitish -confusing -shipbuilding -relies -surgeons -landings -ravi -baku -moor -suffix -alejandro -##yana -litre -upheld -##unk -rajasthan -##rek -coaster -insists -posture -scenarios -etienne -favoured -appoint -transgender -elephants -poked -greenwood -defences -fulfilled -militant -somali -1758 -chalk -potent -##ucci -migrants -wink -assistants -nos -restriction -activism -niger -##ario -colon -shaun -##sat -daphne -##erated -swam -congregations -reprise -considerations -magnet -playable -xvi -##р -overthrow -tobias -knob -chavez -coding -##mers -propped -katrina -orient -newcomer -##suke -temperate -##pool -farmhouse -interrogation -##vd -committing -##vert -forthcoming -strawberry -joaquin -macau -ponds -shocking -siberia -##cellular -chant -contributors -##nant -##ologists -sped -absorb -hail -1782 -spared -##hore -barbados -karate -opus -originates -saul -##xie -evergreen -leaped -##rock -correlation -exaggerated -weekday -unification -bump -tracing -brig -afb -pathways -utilizing -##ners -mod -mb -disturbance -kneeling -##stad -##guchi -100th -pune -##thy -decreasing -168 -manipulation -miriam -academia -ecosystem -occupational -rbi -##lem -rift -##14 -rotary -stacked -incorporation -awakening -generators -guerrero -racist -##omy -cyber -derivatives -culminated -allie -annals -panzer -sainte -wikipedia -pops -zu -austro -##vate -algerian -politely -nicholson -mornings -educate -tastes -thrill -dartmouth -##gating -db -##jee -regan -differing -concentrating -choreography -divinity -##media -pledged -alexandre -routing -gregor -madeline -##idal -apocalypse -##hora -gunfire -culminating -elves -fined -liang -lam -programmed -tar -guessing -transparency -gabrielle -##gna -cancellation -flexibility -##lining -accession -shea -stronghold -nets -specializes -##rgan -abused -hasan -sgt -ling -exceeding -##₄ -admiration -supermarket -##ark -photographers -specialised -tilt -resonance -hmm -perfume -380 -sami -threatens -garland -botany -guarding -boiled -greet -puppy -russo -supplier -wilmington -vibrant -vijay -##bius -paralympic -grumbled -paige -faa -licking -margins -hurricanes -##gong -fest -grenade -ripping -##uz -counseling -weigh -##sian -needles -wiltshire -edison -costly -##not -fulton -tramway -redesigned -staffordshire -cache -gasping -watkins -sleepy -candidacy -##group -monkeys -timeline -throbbing -##bid -##sos -berth -uzbekistan -vanderbilt -bothering -overturned -ballots -gem -##iger -sunglasses -subscribers -hooker -compelling -ang -exceptionally -saloon -stab -##rdi -carla -terrifying -rom -##vision -coil -##oids -satisfying -vendors -31st -mackay -deities -overlooked -ambient -bahamas -felipe -olympia -whirled -botanist -advertised -tugging -##dden -disciples -morales -unionist -rites -foley -morse -motives -creepy -##₀ -soo -##sz -bargain -highness -frightening -turnpike -tory -reorganization -##cer -depict -biographer -##walk -unopposed -manifesto -##gles -institut -emile -accidental -kapoor -##dam -kilkenny -cortex -lively -##13 -romanesque -jain -shan -cannons -##ood -##ske -petrol -echoing -amalgamated -disappears -cautious -proposes -sanctions -trenton -##ر -flotilla -aus -contempt -tor -canary -cote -theirs -##hun -conceptual -deleted -fascinating -paso -blazing -elf -honourable -hutchinson -##eiro -##outh -##zin -surveyor -tee -amidst -wooded -reissue -intro -##ono -cobb -shelters -newsletter -hanson -brace -encoding -confiscated -dem -caravan -marino -scroll -melodic -cows -imam -##adi -##aneous -northward -searches -biodiversity -cora -310 -roaring -##bers -connell -theologian -halo -compose -pathetic -unmarried -dynamo -##oot -az -calculation -toulouse -deserves -humour -nr -forgiveness -tam -undergone -martyr -pamela -myths -whore -counselor -hicks -290 -heavens -battleship -electromagnetic -##bbs -stellar -establishments -presley -hopped -##chin -temptation -90s -wills -nas -##yuan -nhs -##nya -seminars -##yev -adaptations -gong -asher -lex -indicator -sikh -tobago -cites -goin -##yte -satirical -##gies -characterised -correspond -bubbles -lure -participates -##vid -eruption -skate -therapeutic -1785 -canals -wholesale -defaulted -sac -460 -petit -##zzled -virgil -leak -ravens -256 -portraying -##yx -ghetto -creators -dams -portray -vicente -##rington -fae -namesake -bounty -##arium -joachim -##ota -##iser -aforementioned -axle -snout -depended -dismantled -reuben -480 -##ibly -gallagher -##lau -##pd -earnest -##ieu -##iary -inflicted -objections -##llar -asa -gritted -##athy -jericho -##sea -##was -flick -underside -ceramics -undead -substituted -195 -eastward -undoubtedly -wheeled -chimney -##iche -guinness -cb -##ager -siding -##bell -traitor -baptiste -disguised -inauguration -149 -tipperary -choreographer -perched -warmed -stationary -eco -##ike -##ntes -bacterial -##aurus -flores -phosphate -##core -attacker -invaders -alvin -intersects -a1 -indirectly -immigrated -businessmen -cornelius -valves -narrated -pill -sober -ul -nationale -monastic -applicants -scenery -##jack -161 -motifs -constitutes -cpu -##osh -jurisdictions -sd -tuning -irritation -woven -##uddin -fertility -gao -##erie -antagonist -impatient -glacial -hides -boarded -denominations -interception -##jas -cookie -nicola -##tee -algebraic -marquess -bahn -parole -buyers -bait -turbines -paperwork -bestowed -natasha -renee -oceans -purchases -157 -vaccine -215 -##tock -fixtures -playhouse -integrate -jai -oswald -intellectuals -##cky -booked -nests -mortimer -##isi -obsession -sept -##gler -##sum -440 -scrutiny -simultaneous -squinted -##shin -collects -oven -shankar -penned -remarkably -##я -slips -luggage -spectral -1786 -collaborations -louie -consolidation -##ailed -##ivating -420 -hoover -blackpool -harness -ignition -vest -tails -belmont -mongol -skinner -##nae -visually -mage -derry -##tism -##unce -stevie -transitional -##rdy -redskins -drying -prep -prospective -##21 -annoyance -oversee -##loaded -fills -##books -##iki -announces -fda -scowled -respects -prasad -mystic -tucson -##vale -revue -springer -bankrupt -1772 -aristotle -salvatore -habsburg -##geny -dal -natal -nut -pod -chewing -darts -moroccan -walkover -rosario -lenin -punjabi -##ße -grossed -scattering -wired -invasive -hui -polynomial -corridors -wakes -gina -portrays -##cratic -arid -retreating -erich -irwin -sniper -##dha -linen -lindsey -maneuver -butch -shutting -socio -bounce -commemorative -postseason -jeremiah -pines -275 -mystical -beads -bp -abbas -furnace -bidding -consulted -assaulted -empirical -rubble -enclosure -sob -weakly -cancel -polly -yielded -##emann -curly -prediction -battered -70s -vhs -jacqueline -render -sails -barked -detailing -grayson -riga -sloane -raging -##yah -herbs -bravo -##athlon -alloy -giggle -imminent -suffers -assumptions -waltz -##itate -accomplishments -##ited -bathing -remixed -deception -prefix -##emia -deepest -##tier -##eis -balkan -frogs -##rong -slab -##pate -philosophers -peterborough -grains -imports -dickinson -rwanda -##atics -1774 -dirk -lan -tablets -##rove -clone -##rice -caretaker -hostilities -mclean -##gre -regimental -treasures -norms -impose -tsar -tango -diplomacy -variously -complain -192 -recognise -arrests -1779 -celestial -pulitzer -##dus -bing -libretto -##moor -adele -splash -##rite -expectation -lds -confronts -##izer -spontaneous -harmful -wedge -entrepreneurs -buyer -##ope -bilingual -translate -rugged -conner -circulated -uae -eaton -##gra -##zzle -lingered -lockheed -vishnu -reelection -alonso -##oom -joints -yankee -headline -cooperate -heinz -laureate -invading -##sford -echoes -scandinavian -##dham -hugging -vitamin -salute -micah -hind -trader -##sper -radioactive -##ndra -militants -poisoned -ratified -remark -campeonato -deprived -wander -prop -##dong -outlook -##tani -##rix -##eye -chiang -darcy -##oping -mandolin -spice -statesman -babylon -182 -walled -forgetting -afro -##cap -158 -giorgio -buffer -##polis -planetary -##gis -overlap -terminals -kinda -centenary -##bir -arising -manipulate -elm -ke -1770 -ak -##tad -chrysler -mapped -moose -pomeranian -quad -macarthur -assemblies -shoreline -recalls -stratford -##rted -noticeable -##evic -imp -##rita -##sque -accustomed -supplying -tents -disgusted -vogue -sipped -filters -khz -reno -selecting -luftwaffe -mcmahon -tyne -masterpiece -carriages -collided -dunes -exercised -flare -remembers -muzzle -##mobile -heck -##rson -burgess -lunged -middleton -boycott -bilateral -##sity -hazardous -lumpur -multiplayer -spotlight -jackets -goldman -liege -porcelain -rag -waterford -benz -attracts -hopeful -battling -ottomans -kensington -baked -hymns -cheyenne -lattice -levine -borrow -polymer -clashes -michaels -monitored -commitments -denounced -##25 -##von -cavity -##oney -hobby -akin -##holders -futures -intricate -cornish -patty -##oned -illegally -dolphin -##lag -barlow -yellowish -maddie -apologized -luton -plagued -##puram -nana -##rds -sway -fanny -łodz -##rino -psi -suspicions -hanged -##eding -initiate -charlton -##por -nak -competent -235 -analytical -annex -wardrobe -reservations -##rma -sect -162 -fairfax -hedge -piled -buckingham -uneven -bauer -simplicity -snyder -interpret -accountability -donors -moderately -byrd -continents -##cite -##max -disciple -hr -jamaican -ping -nominees -##uss -mongolian -diver -attackers -eagerly -ideological -pillows -miracles -apartheid -revolver -sulfur -clinics -moran -163 -##enko -ile -katy -rhetoric -##icated -chronology -recycling -##hrer -elongated -mughal -pascal -profiles -vibration -databases -domination -##fare -##rant -matthias -digest -rehearsal -polling -weiss -initiation -reeves -clinging -flourished -impress -ngo -##hoff -##ume -buckley -symposium -rhythms -weed -emphasize -transforming -##taking -##gence -##yman -accountant -analyze -flicker -foil -priesthood -voluntarily -decreases -##80 -##hya -slater -sv -charting -mcgill -##lde -moreno -##iu -besieged -zur -robes -##phic -admitting -api -deported -turmoil -peyton -earthquakes -##ares -nationalists -beau -clair -brethren -interrupt -welch -curated -galerie -requesting -164 -##ested -impending -steward -viper -##vina -complaining -beautifully -brandy -foam -nl -1660 -##cake -alessandro -punches -laced -explanations -##lim -attribute -clit -reggie -discomfort -##cards -smoothed -whales -##cene -adler -countered -duffy -disciplinary -widening -recipe -reliance -conducts -goats -gradient -preaching -##shaw -matilda -quasi -striped -meridian -cannabis -cordoba -certificates -##agh -##tering -graffiti -hangs -pilgrims -repeats -##ych -revive -urine -etat -##hawk -fueled -belts -fuzzy -susceptible -##hang -mauritius -salle -sincere -beers -hooks -##cki -arbitration -entrusted -advise -sniffed -seminar -junk -donnell -processors -principality -strapped -celia -mendoza -everton -fortunes -prejudice -starving -reassigned -steamer -##lund -tuck -evenly -foreman -##ffen -dans -375 -envisioned -slit -##xy -baseman -liberia -rosemary -##weed -electrified -periodically -potassium -stride -contexts -sperm -slade -mariners -influx -bianca -subcommittee -##rane -spilling -icao -estuary -##nock -delivers -iphone -##ulata -isa -mira -bohemian -dessert -##sbury -welcoming -proudly -slowing -##chs -musee -ascension -russ -##vian -waits -##psy -africans -exploit -##morphic -gov -eccentric -crab -peck -##ull -entrances -formidable -marketplace -groom -bolted -metabolism -patton -robbins -courier -payload -endure -##ifier -andes -refrigerator -##pr -ornate -##uca -ruthless -illegitimate -masonry -strasbourg -bikes -adobe -##³ -apples -quintet -willingly -niche -bakery -corpses -energetic -##cliffe -##sser -##ards -177 -centimeters -centro -fuscous -cretaceous -rancho -##yde -andrei -telecom -tottenham -oasis -ordination -vulnerability -presiding -corey -cp -penguins -sims -##pis -malawi -piss -##48 -correction -##cked -##ffle -##ryn -countdown -detectives -psychiatrist -psychedelic -dinosaurs -blouse -##get -choi -vowed -##oz -randomly -##pol -49ers -scrub -blanche -bruins -dusseldorf -##using -unwanted -##ums -212 -dominique -elevations -headlights -om -laguna -##oga -1750 -famously -ignorance -shrewsbury -##aine -ajax -breuning -che -confederacy -greco -overhaul -##screen -paz -skirts -disagreement -cruelty -jagged -phoebe -shifter -hovered -viruses -##wes -mandy -##lined -##gc -landlord -squirrel -dashed -##ι -ornamental -gag -wally -grange -literal -spurs -undisclosed -proceeding -yin -##text -billie -orphan -spanned -humidity -indy -weighted -presentations -explosions -lucian -##tary -vaughn -hindus -##anga -##hell -psycho -171 -daytona -protects -efficiently -rematch -sly -tandem -##oya -rebranded -impaired -hee -metropolis -peach -godfrey -diaspora -ethnicity -prosperous -gleaming -dar -grossing -playback -##rden -stripe -pistols -##tain -births -labelled -##cating -172 -rudy -alba -##onne -aquarium -hostility -##gb -##tase -shudder -sumatra -hardest -lakers -consonant -creeping -demos -homicide -capsule -zeke -liberties -expulsion -pueblo -##comb -trait -transporting -##ddin -##neck -##yna -depart -gregg -mold -ledge -hangar -oldham -playboy -termination -analysts -gmbh -romero -##itic -insist -cradle -filthy -brightness -slash -shootout -deposed -bordering -##truct -isis -microwave -tumbled -sheltered -cathy -werewolves -messy -andersen -convex -clapped -clinched -satire -wasting -edo -vc -rufus -##jak -mont -##etti -poznan -##keeping -restructuring -transverse -##rland -azerbaijani -slovene -gestures -roommate -choking -shear -##quist -vanguard -oblivious -##hiro -disagreed -baptism -##lich -coliseum -##aceae -salvage -societe -cory -locke -relocation -relying -versailles -ahl -swelling -##elo -cheerful -##word -##edes -gin -sarajevo -obstacle -diverted -##nac -messed -thoroughbred -fluttered -utrecht -chewed -acquaintance -assassins -dispatch -mirza -##wart -nike -salzburg -swell -yen -##gee -idle -ligue -samson -##nds -##igh -playful -spawned -##cise -tease -##case -burgundy -##bot -stirring -skeptical -interceptions -marathi -##dies -bedrooms -aroused -pinch -##lik -preferences -tattoos -buster -digitally -projecting -rust -##ital -kitten -priorities -addison -pseudo -##guard -dusk -icons -sermon -##psis -##iba -bt -##lift -##xt -ju -truce -rink -##dah -##wy -defects -psychiatry -offences -calculate -glucose -##iful -##rized -##unda -francaise -##hari -richest -warwickshire -carly -1763 -purity -redemption -lending -##cious -muse -bruises -cerebral -aero -carving -##name -preface -terminology -invade -monty -##int -anarchist -blurred -##iled -rossi -treats -guts -shu -foothills -ballads -undertaking -premise -cecilia -affiliates -blasted -conditional -wilder -minors -drone -rudolph -buffy -swallowing -horton -attested -##hop -rutherford -howell -primetime -livery -penal -##bis -minimize -hydro -wrecked -wrought -palazzo -##gling -cans -vernacular -friedman -nobleman -shale -walnut -danielle -##ection -##tley -sears -##kumar -chords -lend -flipping -streamed -por -dracula -gallons -sacrifices -gamble -orphanage -##iman -mckenzie -##gible -boxers -daly -##balls -##ان -208 -##ific -##rative -##iq -exploited -slated -##uity -circling -hillary -pinched -goldberg -provost -campaigning -lim -piles -ironically -jong -mohan -successors -usaf -##tem -##ught -autobiographical -haute -preserves -##ending -acquitted -comparisons -203 -hydroelectric -gangs -cypriot -torpedoes -rushes -chrome -derive -bumps -instability -fiat -pets -##mbe -silas -dye -reckless -settler -##itation -info -heats -##writing -176 -canonical -maltese -fins -mushroom -stacy -aspen -avid -##kur -##loading -vickers -gaston -hillside -statutes -wilde -gail -kung -sabine -comfortably -motorcycles -##rgo -169 -pneumonia -fetch -##sonic -axel -faintly -parallels -##oop -mclaren -spouse -compton -interdisciplinary -miner -##eni -181 -clamped -##chal -##llah -separates -versa -##mler -scarborough -labrador -##lity -##osing -rutgers -hurdles -como -166 -burt -divers -##100 -wichita -cade -coincided -##erson -bruised -mla -##pper -vineyard -##ili -##brush -notch -mentioning -jase -hearted -kits -doe -##acle -pomerania -##ady -ronan -seizure -pavel -problematic -##zaki -domenico -##ulin -catering -penelope -dependence -parental -emilio -ministerial -atkinson -##bolic -clarkson -chargers -colby -grill -peeked -arises -summon -##aged -fools -##grapher -faculties -qaeda -##vial -garner -refurbished -##hwa -geelong -disasters -nudged -bs -shareholder -lori -algae -reinstated -rot -##ades -##nous -invites -stainless -183 -inclusive -##itude -diocesan -til -##icz -denomination -##xa -benton -floral -registers -##ider -##erman -##kell -absurd -brunei -guangzhou -hitter -retaliation -##uled -##eve -blanc -nh -consistency -contamination -##eres -##rner -dire -palermo -broadcasters -diaries -inspire -vols -brewer -tightening -ky -mixtape -hormone -##tok -stokes -##color -##dly -##ssi -pg -##ometer -##lington -sanitation -##tility -intercontinental -apps -##adt -¹⁄₂ -cylinders -economies -favourable -unison -croix -gertrude -odyssey -vanity -dangling -##logists -upgrades -dice -middleweight -practitioner -##ight -206 -henrik -parlor -orion -angered -lac -python -blurted -##rri -sensual -intends -swings -angled -##phs -husky -attain -peerage -precinct -textiles -cheltenham -shuffled -dai -confess -tasting -bhutan -##riation -tyrone -segregation -abrupt -ruiz -##rish -smirked -blackwell -confidential -browning -amounted -##put -vase -scarce -fabulous -raided -staple -guyana -unemployed -glider -shay -##tow -carmine -troll -intervene -squash -superstar -##uce -cylindrical -len -roadway -researched -handy -##rium -##jana -meta -lao -declares -##rring -##tadt -##elin -##kova -willem -shrubs -napoleonic -realms -skater -qi -volkswagen -##ł -tad -hara -archaeologist -awkwardly -eerie -##kind -wiley -##heimer -##24 -titus -organizers -cfl -crusaders -lama -usb -vent -enraged -thankful -occupants -maximilian -##gaard -possessing -textbooks -##oran -collaborator -quaker -##ulo -avalanche -mono -silky -straits -isaiah -mustang -surged -resolutions -potomac -descend -cl -kilograms -plato -strains -saturdays -##olin -bernstein -##ype -holstein -ponytail -##watch -belize -conversely -heroine -perpetual -##ylus -charcoal -piedmont -glee -negotiating -backdrop -prologue -##jah -##mmy -pasadena -climbs -ramos -sunni -##holm -##tner -##tri -anand -deficiency -hertfordshire -stout -##avi -aperture -orioles -##irs -doncaster -intrigued -bombed -coating -otis -##mat -cocktail -##jit -##eto -amir -arousal -sar -##proof -##act -##ories -dixie -pots -##bow -whereabouts -159 -##fted -drains -bullying -cottages -scripture -coherent -fore -poe -appetite -##uration -sampled -##ators -##dp -derrick -rotor -jays -peacock -installment -##rro -advisors -##coming -rodeo -scotch -##mot -##db -##fen -##vant -ensued -rodrigo -dictatorship -martyrs -twenties -##н -towed -incidence -marta -rainforest -sai -scaled -##cles -oceanic -qualifiers -symphonic -mcbride -dislike -generalized -aubrey -colonization -##iation -##lion -##ssing -disliked -lublin -salesman -##ulates -spherical -whatsoever -sweating -avalon -contention -punt -severity -alderman -atari -##dina -##grant -##rop -scarf -seville -vertices -annexation -fairfield -fascination -inspiring -launches -palatinate -regretted -##rca -feral -##iom -elk -nap -olsen -reddy -yong -##leader -##iae -garment -transports -feng -gracie -outrage -viceroy -insides -##esis -breakup -grady -organizer -softer -grimaced -222 -murals -galicia -arranging -vectors -##rsten -bas -##sb -##cens -sloan -##eka -bitten -ara -fender -nausea -bumped -kris -banquet -comrades -detector -persisted -##llan -adjustment -endowed -cinemas -##shot -sellers -##uman -peek -epa -kindly -neglect -simpsons -talon -mausoleum -runaway -hangul -lookout -##cic -rewards -coughed -acquainted -chloride -##ald -quicker -accordion -neolithic -##qa -artemis -coefficient -lenny -pandora -tx -##xed -ecstasy -litter -segunda -chairperson -gemma -hiss -rumor -vow -nasal -antioch -compensate -patiently -transformers -##eded -judo -morrow -penis -posthumous -philips -bandits -husbands -denote -flaming -##any -##phones -langley -yorker -1760 -walters -##uo -##kle -gubernatorial -fatty -samsung -leroy -outlaw -##nine -unpublished -poole -jakob -##ᵢ -##ₙ -crete -distorted -superiority -##dhi -intercept -crust -mig -claus -crashes -positioning -188 -stallion -301 -frontal -armistice -##estinal -elton -aj -encompassing -camel -commemorated -malaria -woodward -calf -cigar -penetrate -##oso -willard -##rno -##uche -illustrate -amusing -convergence -noteworthy -##lma -##rva -journeys -realise -manfred -##sable -410 -##vocation -hearings -fiance -##posed -educators -provoked -adjusting -##cturing -modular -stockton -paterson -vlad -rejects -electors -selena -maureen -##tres -uber -##rce -swirled -##num -proportions -nanny -pawn -naturalist -parma -apostles -awoke -ethel -wen -##bey -monsoon -overview -##inating -mccain -rendition -risky -adorned -##ih -equestrian -germain -nj -conspicuous -confirming -##yoshi -shivering -##imeter -milestone -rumours -flinched -bounds -smacked -token -##bei -lectured -automobiles -##shore -impacted -##iable -nouns -nero -##leaf -ismail -prostitute -trams -##lace -bridget -sud -stimulus -impressions -reins -revolves -##oud -##gned -giro -honeymoon -##swell -criterion -##sms -##uil -libyan -prefers -##osition -211 -preview -sucks -accusation -bursts -metaphor -diffusion -tolerate -faye -betting -cinematographer -liturgical -specials -bitterly -humboldt -##ckle -flux -rattled -##itzer -archaeologists -odor -authorised -marshes -discretion -##ов -alarmed -archaic -inverse -##leton -explorers -##pine -drummond -tsunami -woodlands -##minate -##tland -booklet -insanity -owning -insert -crafted -calculus -##tore -receivers -##bt -stung -##eca -##nched -prevailing -travellers -eyeing -lila -graphs -##borne -178 -julien -##won -morale -adaptive -therapist -erica -cw -libertarian -bowman -pitches -vita -##ional -crook -##ads -##entation -caledonia -mutiny -##sible -1840s -automation -##ß -flock -##pia -ironic -pathology -##imus -remarried -##22 -joker -withstand -energies -##att -shropshire -hostages -madeleine -tentatively -conflicting -mateo -recipes -euros -ol -mercenaries -nico -##ndon -albuquerque -augmented -mythical -bel -freud -##child -cough -##lica -365 -freddy -lillian -genetically -nuremberg -calder -209 -bonn -outdoors -paste -suns -urgency -vin -restraint -tyson -##cera -##selle -barrage -bethlehem -kahn -##par -mounts -nippon -barony -happier -ryu -makeshift -sheldon -blushed -castillo -barking -listener -taped -bethel -fluent -headlines -pornography -rum -disclosure -sighing -mace -doubling -gunther -manly -##plex -rt -interventions -physiological -forwards -emerges -##tooth -##gny -compliment -rib -recession -visibly -barge -faults -connector -exquisite -prefect -##rlin -patio -##cured -elevators -brandt -italics -pena -173 -wasp -satin -ea -botswana -graceful -respectable -##jima -##rter -##oic -franciscan -generates -##dl -alfredo -disgusting -##olate -##iously -sherwood -warns -cod -promo -cheryl -sino -##ة -##escu -twitch -##zhi -brownish -thom -ortiz -##dron -densely -##beat -carmel -reinforce -##bana -187 -anastasia -downhill -vertex -contaminated -remembrance -harmonic -homework -##sol -fiancee -gears -olds -angelica -loft -ramsay -quiz -colliery -sevens -##cape -autism -##hil -walkway -##boats -ruben -abnormal -ounce -khmer -##bbe -zachary -bedside -morphology -punching -##olar -sparrow -convinces -##35 -hewitt -queer -remastered -rods -mabel -solemn -notified -lyricist -symmetric -##xide -174 -encore -passports -wildcats -##uni -baja -##pac -mildly -##ease -bleed -commodity -mounds -glossy -orchestras -##omo -damian -prelude -ambitions -##vet -awhile -remotely -##aud -asserts -imply -##iques -distinctly -modelling -remedy -##dded -windshield -dani -xiao -##endra -audible -powerplant -1300 -invalid -elemental -acquisitions -##hala -immaculate -libby -plata -smuggling -ventilation -denoted -minh -##morphism -430 -differed -dion -kelley -lore -mocking -sabbath -spikes -hygiene -drown -runoff -stylized -tally -liberated -aux -interpreter -righteous -aba -siren -reaper -pearce -millie -##cier -##yra -gaius -##iso -captures -##ttering -dorm -claudio -##sic -benches -knighted -blackness -##ored -discount -fumble -oxidation -routed -##ς -novak -perpendicular -spoiled -fracture -splits -##urt -pads -topology -##cats -axes -fortunate -offenders -protestants -esteem -221 -broadband -convened -frankly -hound -prototypes -isil -facilitated -keel -##sher -sahara -awaited -bubba -orb -prosecutors -186 -hem -520 -##xing -relaxing -remnant -romney -sorted -slalom -stefano -ulrich -##active -exemption -folder -pauses -foliage -hitchcock -epithet -204 -criticisms -##aca -ballistic -brody -hinduism -chaotic -youths -equals -##pala -pts -thicker -analogous -capitalist -improvised -overseeing -sinatra -ascended -beverage -##tl -straightforward -##kon -curran -##west -bois -325 -induce -surveying -emperors -sax -unpopular -##kk -cartoonist -fused -##mble -unto -##yuki -localities -##cko -##ln -darlington -slain -academie -lobbying -sediment -puzzles -##grass -defiance -dickens -manifest -tongues -alumnus -arbor -coincide -184 -appalachian -mustafa -examiner -cabaret -traumatic -yves -bracelet -draining -heroin -magnum -baths -odessa -consonants -mitsubishi -##gua -kellan -vaudeville -##fr -joked -null -straps -probation -##ław -ceded -interfaces -##pas -##zawa -blinding -viet -224 -rothschild -museo -640 -huddersfield -##vr -tactic -##storm -brackets -dazed -incorrectly -##vu -reg -glazed -fearful -manifold -benefited -irony -##sun -stumbling -##rte -willingness -balkans -mei -wraps -##aba -injected -##lea -gu -syed -harmless -##hammer -bray -takeoff -poppy -timor -cardboard -astronaut -purdue -weeping -southbound -cursing -stalls -diagonal -##neer -lamar -bryce -comte -weekdays -harrington -##uba -negatively -##see -lays -grouping -##cken -##henko -affirmed -halle -modernist -##lai -hodges -smelling -aristocratic -baptized -dismiss -justification -oilers -##now -coupling -qin -snack -healer -##qing -gardener -layla -battled -formulated -stephenson -gravitational -##gill -##jun -1768 -granny -coordinating -suites -##cd -##ioned -monarchs -##cote -##hips -sep -blended -apr -barrister -deposition -fia -mina -policemen -paranoid -##pressed -churchyard -covert -crumpled -creep -abandoning -tr -transmit -conceal -barr -understands -readiness -spire -##cology -##enia -##erry -610 -startling -unlock -vida -bowled -slots -##nat -##islav -spaced -trusting -admire -rig -##ink -slack -##70 -mv -207 -casualty -##wei -classmates -##odes -##rar -##rked -amherst -furnished -evolve -foundry -menace -mead -##lein -flu -wesleyan -##kled -monterey -webber -##vos -wil -##mith -##на -bartholomew -justices -restrained -##cke -amenities -191 -mediated -sewage -trenches -ml -mainz -##thus -1800s -##cula -##inski -caine -bonding -213 -converts -spheres -superseded -marianne -crypt -sweaty -ensign -historia -##br -spruce -##post -##ask -forks -thoughtfully -yukon -pamphlet -ames -##uter -karma -##yya -bryn -negotiation -sighs -incapable -##mbre -##ntial -actresses -taft -##mill -luce -prevailed -##amine -1773 -motionless -envoy -testify -investing -sculpted -instructors -provence -kali -cullen -horseback -##while -goodwin -##jos -gaa -norte -##ldon -modify -wavelength -abd -214 -skinned -sprinter -forecast -scheduling -marries -squared -tentative -##chman -boer -##isch -bolts -swap -fisherman -assyrian -impatiently -guthrie -martins -murdoch -194 -tanya -nicely -dolly -lacy -med -##45 -syn -decks -fashionable -millionaire -##ust -surfing -##ml -##ision -heaved -tammy -consulate -attendees -routinely -197 -fuse -saxophonist -backseat -malaya -##lord -scowl -tau -##ishly -193 -sighted -steaming -##rks -303 -911 -##holes -##hong -ching -##wife -bless -conserved -jurassic -stacey -unix -zion -chunk -rigorous -blaine -198 -peabody -slayer -dismay -brewers -nz -##jer -det -##glia -glover -postwar -int -penetration -sylvester -imitation -vertically -airlift -heiress -knoxville -viva -##uin -390 -macon -##rim -##fighter -##gonal -janice -##orescence -##wari -marius -belongings -leicestershire -196 -blanco -inverted -preseason -sanity -sobbing -##due -##elt -##dled -collingwood -regeneration -flickering -shortest -##mount -##osi -feminism -##lat -sherlock -cabinets -fumbled -northbound -precedent -snaps -##mme -researching -##akes -guillaume -insights -manipulated -vapor -neighbour -sap -gangster -frey -f1 -stalking -scarcely -callie -barnett -tendencies -audi -doomed -assessing -slung -panchayat -ambiguous -bartlett -##etto -distributing -violating -wolverhampton -##hetic -swami -histoire -##urus -liable -pounder -groin -hussain -larsen -popping -surprises -##atter -vie -curt -##station -mute -relocate -musicals -authorization -richter -##sef -immortality -tna -bombings -##press -deteriorated -yiddish -##acious -robbed -colchester -cs -pmid -ao -verified -balancing -apostle -swayed -recognizable -oxfordshire -retention -nottinghamshire -contender -judd -invitational -shrimp -uhf -##icient -cleaner -longitudinal -tanker -##mur -acronym -broker -koppen -sundance -suppliers -##gil -4000 -clipped -fuels -petite -##anne -landslide -helene -diversion -populous -landowners -auspices -melville -quantitative -##xes -ferries -nicky -##llus -doo -haunting -roche -carver -downed -unavailable -##pathy -approximation -hiroshima -##hue -garfield -valle -comparatively -keyboardist -traveler -##eit -congestion -calculating -subsidiaries -##bate -serb -modernization -fairies -deepened -ville -averages -##lore -inflammatory -tonga -##itch -co₂ -squads -##hea -gigantic -serum -enjoyment -retailer -verona -35th -cis -##phobic -magna -technicians -##vati -arithmetic -##sport -levin -##dation -amtrak -chow -sienna -##eyer -backstage -entrepreneurship -##otic -learnt -tao -##udy -worcestershire -formulation -baggage -hesitant -bali -sabotage -##kari -barren -enhancing -murmur -pl -freshly -putnam -syntax -aces -medicines -resentment -bandwidth -##sier -grins -chili -guido -##sei -framing -implying -gareth -lissa -genevieve -pertaining -admissions -geo -thorpe -proliferation -sato -bela -analyzing -parting -##gor -awakened -##isman -huddled -secrecy -##kling -hush -gentry -540 -dungeons -##ego -coasts -##utz -sacrificed -##chule -landowner -mutually -prevalence -programmer -adolescent -disrupted -seaside -gee -trusts -vamp -georgie -##nesian -##iol -schedules -sindh -##market -etched -hm -sparse -bey -beaux -scratching -gliding -unidentified -216 -collaborating -gems -jesuits -oro -accumulation -shaping -mbe -anal -##xin -231 -enthusiasts -newscast -##egan -janata -dewey -parkinson -179 -ankara -biennial -towering -dd -inconsistent -950 -##chet -thriving -terminate -cabins -furiously -eats -advocating -donkey -marley -muster -phyllis -leiden -##user -grassland -glittering -iucn -loneliness -217 -memorandum -armenians -##ddle -popularized -rhodesia -60s -lame -##illon -sans -bikini -header -orbits -##xx -##finger -##ulator -sharif -spines -biotechnology -strolled -naughty -yates -##wire -fremantle -milo -##mour -abducted -removes -##atin -humming -wonderland -##chrome -##ester -hume -pivotal -##rates -armand -grams -believers -elector -rte -apron -bis -scraped -##yria -endorsement -initials -##llation -eps -dotted -hints -buzzing -emigration -nearer -##tom -indicators -##ulu -coarse -neutron -protectorate -##uze -directional -exploits -pains -loire -1830s -proponents -guggenheim -rabbits -ritchie -305 -hectare -inputs -hutton -##raz -verify -##ako -boilers -longitude -##lev -skeletal -yer -emilia -citrus -compromised -##gau -pokemon -prescription -paragraph -eduard -cadillac -attire -categorized -kenyan -weddings -charley -##bourg -entertain -monmouth -##lles -nutrients -davey -mesh -incentive -practised -ecosystems -kemp -subdued -overheard -##rya -bodily -maxim -##nius -apprenticeship -ursula -##fight -lodged -rug -silesian -unconstitutional -patel -inspected -coyote -unbeaten -##hak -34th -disruption -convict -parcel -##cl -##nham -collier -implicated -mallory -##iac -##lab -susannah -winkler -##rber -shia -phelps -sediments -graphical -robotic -##sner -adulthood -mart -smoked -##isto -kathryn -clarified -##aran -divides -convictions -oppression -pausing -burying -##mt -federico -mathias -eileen -##tana -kite -hunched -##acies -189 -##atz -disadvantage -liza -kinetic -greedy -paradox -yokohama -dowager -trunks -ventured -##gement -gupta -vilnius -olaf -##thest -crimean -hopper -##ej -progressively -arturo -mouthed -arrondissement -##fusion -rubin -simulcast -oceania -##orum -##stra -##rred -busiest -intensely -navigator -cary -##vine -##hini -##bies -fife -rowe -rowland -posing -insurgents -shafts -lawsuits -activate -conor -inward -culturally -garlic -265 -##eering -eclectic -##hui -##kee -##nl -furrowed -vargas -meteorological -rendezvous -##aus -culinary -commencement -##dition -quota -##notes -mommy -salaries -overlapping -mule -##iology -##mology -sums -wentworth -##isk -##zione -mainline -subgroup -##illy -hack -plaintiff -verdi -bulb -differentiation -engagements -multinational -supplemented -bertrand -caller -regis -##naire -##sler -##arts -##imated -blossom -propagation -kilometer -viaduct -vineyards -##uate -beckett -optimization -golfer -songwriters -seminal -semitic -thud -volatile -evolving -ridley -##wley -trivial -distributions -scandinavia -jiang -##ject -wrestled -insistence -##dio -emphasizes -napkin -##ods -adjunct -rhyme -##ricted -##eti -hopeless -surrounds -tremble -32nd -smoky -##ntly -oils -medicinal -padded -steer -wilkes -219 -255 -concessions -hue -uniquely -blinded -landon -yahoo -##lane -hendrix -commemorating -dex -specify -chicks -##ggio -intercity -1400 -morley -##torm -highlighting -##oting -pang -oblique -stalled -##liner -flirting -newborn -1769 -bishopric -shaved -232 -currie -##ush -dharma -spartan -##ooped -favorites -smug -novella -sirens -abusive -creations -espana -##lage -paradigm -semiconductor -sheen -##rdo -##yen -##zak -nrl -renew -##pose -##tur -adjutant -marches -norma -##enity -ineffective -weimar -grunt -##gat -lordship -plotting -expenditure -infringement -lbs -refrain -av -mimi -mistakenly -postmaster -1771 -##bara -ras -motorsports -tito -199 -subjective -##zza -bully -stew -##kaya -prescott -1a -##raphic -##zam -bids -styling -paranormal -reeve -sneaking -exploding -katz -akbar -migrant -syllables -indefinitely -##ogical -destroys -replaces -applause -##phine -pest -##fide -218 -articulated -bertie -##thing -##cars -##ptic -courtroom -crowley -aesthetics -cummings -tehsil -hormones -titanic -dangerously -##ibe -stadion -jaenelle -auguste -ciudad -##chu -mysore -partisans -##sio -lucan -philipp -##aly -debating -henley -interiors -##rano -##tious -homecoming -beyonce -usher -henrietta -prepares -weeds -##oman -ely -plucked -##pire -##dable -luxurious -##aq -artifact -password -pasture -juno -maddy -minsk -##dder -##ologies -##rone -assessments -martian -royalist -1765 -examines -##mani -##rge -nino -223 -parry -scooped -relativity -##eli -##uting -##cao -congregational -noisy -traverse -##agawa -strikeouts -nickelodeon -obituary -transylvania -binds -depictions -polk -trolley -##yed -##lard -breeders -##under -dryly -hokkaido -1762 -strengths -stacks -bonaparte -connectivity -neared -prostitutes -stamped -anaheim -gutierrez -sinai -##zzling -bram -fresno -madhya -##86 -proton -##lena -##llum -##phon -reelected -wanda -##anus -##lb -ample -distinguishing -##yler -grasping -sermons -tomato -bland -stimulation -avenues -##eux -spreads -scarlett -fern -pentagon -assert -baird -chesapeake -ir -calmed -distortion -fatalities -##olis -correctional -pricing -##astic -##gina -prom -dammit -ying -collaborate -##chia -welterweight -33rd -pointer -substitution -bonded -umpire -communicating -multitude -paddle -##obe -federally -intimacy -##insky -betray -ssr -##lett -##lean -##lves -##therapy -airbus -##tery -functioned -ud -bearer -biomedical -netflix -##hire -##nca -condom -brink -ik -##nical -macy -##bet -flap -gma -experimented -jelly -lavender -##icles -##ulia -munro -##mian -##tial -rye -##rle -60th -gigs -hottest -rotated -predictions -fuji -bu -##erence -##omi -barangay -##fulness -##sas -clocks -##rwood -##liness -cereal -roe -wight -decker -uttered -babu -onion -xml -forcibly -##df -petra -sarcasm -hartley -peeled -storytelling -##42 -##xley -##ysis -##ffa -fibre -kiel -auditor -fig -harald -greenville -##berries -geographically -nell -quartz -##athic -cemeteries -##lr -crossings -nah -holloway -reptiles -chun -sichuan -snowy -660 -corrections -##ivo -zheng -ambassadors -blacksmith -fielded -fluids -hardcover -turnover -medications -melvin -academies -##erton -ro -roach -absorbing -spaniards -colton -##founded -outsider -espionage -kelsey -245 -edible -##ulf -dora -establishes -##sham -##tries -contracting -##tania -cinematic -costello -nesting -##uron -connolly -duff -##nology -mma -##mata -fergus -sexes -gi -optics -spectator -woodstock -banning -##hee -##fle -differentiate -outfielder -refinery -226 -312 -gerhard -horde -lair -drastically -##udi -landfall -##cheng -motorsport -odi -##achi -predominant -quay -skins -##ental -edna -harshly -complementary -murdering -##aves -wreckage -##90 -ono -outstretched -lennox -munitions -galen -reconcile -470 -scalp -bicycles -gillespie -questionable -rosenberg -guillermo -hostel -jarvis -kabul -volvo -opium -yd -##twined -abuses -decca -outpost -##cino -sensible -neutrality -##64 -ponce -anchorage -atkins -turrets -inadvertently -disagree -libre -vodka -reassuring -weighs -##yal -glide -jumper -ceilings -repertory -outs -stain -##bial -envy -##ucible -smashing -heightened -policing -hyun -mixes -lai -prima -##ples -celeste -##bina -lucrative -intervened -kc -manually -##rned -stature -staffed -bun -bastards -nairobi -priced -##auer -thatcher -##kia -tripped -comune -##ogan -##pled -brasil -incentives -emanuel -hereford -musica -##kim -benedictine -biennale -##lani -eureka -gardiner -rb -knocks -sha -##ael -##elled -##onate -efficacy -ventura -masonic -sanford -maize -leverage -##feit -capacities -santana -##aur -novelty -vanilla -##cter -##tour -benin -##oir -##rain -neptune -drafting -tallinn -##cable -humiliation -##boarding -schleswig -fabian -bernardo -liturgy -spectacle -sweeney -pont -routledge -##tment -cosmos -ut -hilt -sleek -universally -##eville -##gawa -typed -##dry -favors -allegheny -glaciers -##rly -recalling -aziz -##log -parasite -requiem -auf -##berto -##llin -illumination -##breaker -##issa -festivities -bows -govern -vibe -vp -333 -sprawled -larson -pilgrim -bwf -leaping -##rts -##ssel -alexei -greyhound -hoarse -##dler -##oration -seneca -##cule -gaping -##ulously -##pura -cinnamon -##gens -##rricular -craven -fantasies -houghton -engined -reigned -dictator -supervising -##oris -bogota -commentaries -unnatural -fingernails -spirituality -tighten -##tm -canadiens -protesting -intentional -cheers -sparta -##ytic -##iere -##zine -widen -belgarath -controllers -dodd -iaaf -navarre -##ication -defect -squire -steiner -whisky -##mins -560 -inevitably -tome -##gold -chew -##uid -##lid -elastic -##aby -streaked -alliances -jailed -regal -##ined -##phy -czechoslovak -narration -absently -##uld -bluegrass -guangdong -quran -criticizing -hose -hari -##liest -##owa -skier -streaks -deploy -##lom -raft -bose -dialed -huff -##eira -haifa -simplest -bursting -endings -ib -sultanate -##titled -franks -whitman -ensures -sven -##ggs -collaborators -forster -organising -ui -banished -napier -injustice -teller -layered -thump -##otti -roc -battleships -evidenced -fugitive -sadie -robotics -##roud -equatorial -geologist -##iza -yielding -##bron -##sr -internationale -mecca -##diment -sbs -skyline -toad -uploaded -reflective -undrafted -lal -leafs -bayern -##dai -lakshmi -shortlisted -##stick -##wicz -camouflage -donate -af -christi -lau -##acio -disclosed -nemesis -1761 -assemble -straining -northamptonshire -tal -##asi -bernardino -premature -heidi -42nd -coefficients -galactic -reproduce -buzzed -sensations -zionist -monsieur -myrtle -##eme -archery -strangled -musically -viewpoint -antiquities -bei -trailers -seahawks -cured -pee -preferring -tasmanian -lange -sul -##mail -##working -colder -overland -lucivar -massey -gatherings -haitian -##smith -disapproval -flaws -##cco -##enbach -1766 -npr -##icular -boroughs -creole -forums -techno -1755 -dent -abdominal -streetcar -##eson -##stream -procurement -gemini -predictable -##tya -acheron -christoph -feeder -fronts -vendor -bernhard -jammu -tumors -slang -##uber -goaltender -twists -curving -manson -vuelta -mer -peanut -confessions -pouch -unpredictable -allowance -theodor -vascular -##factory -bala -authenticity -metabolic -coughing -nanjing -##cea -pembroke -##bard -splendid -36th -ff -hourly -##ahu -elmer -handel -##ivate -awarding -thrusting -dl -experimentation -##hesion -##46 -caressed -entertained -steak -##rangle -biologist -orphans -baroness -oyster -stepfather -##dridge -mirage -reefs -speeding -##31 -barons -1764 -227 -inhabit -preached -repealed -##tral -honoring -boogie -captives -administer -johanna -##imate -gel -suspiciously -1767 -sobs -##dington -backbone -hayward -garry -##folding -##nesia -maxi -##oof -##ppe -ellison -galileo -##stand -crimea -frenzy -amour -bumper -matrices -natalia -baking -garth -palestinians -##grove -smack -conveyed -ensembles -gardening -##manship -##rup -##stituting -1640 -harvesting -topography -jing -shifters -dormitory -##carriage -##lston -ist -skulls -##stadt -dolores -jewellery -sarawak -##wai -##zier -fences -christy -confinement -tumbling -credibility -fir -stench -##bria -##plication -##nged -##sam -virtues -##belt -marjorie -pba -##eem -##made -celebrates -schooner -agitated -barley -fulfilling -anthropologist -##pro -restrict -novi -regulating -##nent -padres -##rani -##hesive -loyola -tabitha -milky -olson -proprietor -crambidae -guarantees -intercollegiate -ljubljana -hilda -##sko -ignorant -hooded -##lts -sardinia -##lidae -##vation -frontman -privileged -witchcraft -##gp -jammed -laude -poking -##than -bracket -amazement -yunnan -##erus -maharaja -linnaeus -264 -commissioning -milano -peacefully -##logies -akira -rani -regulator -##36 -grasses -##rance -luzon -crows -compiler -gretchen -seaman -edouard -tab -buccaneers -ellington -hamlets -whig -socialists -##anto -directorial -easton -mythological -##kr -##vary -rhineland -semantic -taut -dune -inventions -succeeds -##iter -replication -branched -##pired -jul -prosecuted -kangaroo -penetrated -##avian -middlesbrough -doses -bleak -madam -predatory -relentless -##vili -reluctance -##vir -hailey -crore -silvery -1759 -monstrous -swimmers -transmissions -hawthorn -informing -##eral -toilets -caracas -crouch -kb -##sett -295 -cartel -hadley -##aling -alexia -yvonne -##biology -cinderella -eton -superb -blizzard -stabbing -industrialist -maximus -##gm -##orus -groves -maud -clade -oversized -comedic -##bella -rosen -nomadic -fulham -montane -beverages -galaxies -redundant -swarm -##rot -##folia -##llis -buckinghamshire -fen -bearings -bahadur -##rom -gilles -phased -dynamite -faber -benoit -vip -##ount -##wd -booking -fractured -tailored -anya -spices -westwood -cairns -auditions -inflammation -steamed -##rocity -##acion -##urne -skyla -thereof -watford -torment -archdeacon -transforms -lulu -demeanor -fucked -serge -##sor -mckenna -minas -entertainer -##icide -caress -originate -residue -##sty -1740 -##ilised -##org -beech -##wana -subsidies -##ghton -emptied -gladstone -ru -firefighters -voodoo -##rcle -het -nightingale -tamara -edmond -ingredient -weaknesses -silhouette -285 -compatibility -withdrawing -hampson -##mona -anguish -giggling -##mber -bookstore -##jiang -southernmost -tilting -##vance -bai -economical -rf -briefcase -dreadful -hinted -projections -shattering -totaling -##rogate -analogue -indicted -periodical -fullback -##dman -haynes -##tenberg -##ffs -##ishment -1745 -thirst -stumble -penang -vigorous -##ddling -##kor -##lium -octave -##ove -##enstein -##inen -##ones -siberian -##uti -cbn -repeal -swaying -##vington -khalid -tanaka -unicorn -otago -plastered -lobe -riddle -##rella -perch -##ishing -croydon -filtered -graeme -tripoli -##ossa -crocodile -##chers -sufi -mined -##tung -inferno -lsu -##phi -swelled -utilizes -£2 -cale -periodicals -styx -hike -informally -coop -lund -##tidae -ala -hen -qui -transformations -disposed -sheath -chickens -##cade -fitzroy -sas -silesia -unacceptable -odisha -1650 -sabrina -pe -spokane -ratios -athena -massage -shen -dilemma -##drum -##riz -##hul -corona -doubtful -niall -##pha -##bino -fines -cite -acknowledging -bangor -ballard -bathurst -##resh -huron -mustered -alzheimer -garments -kinase -tyre -warship -##cp -flashback -pulmonary -braun -cheat -kamal -cyclists -constructions -grenades -ndp -traveller -excuses -stomped -signalling -trimmed -futsal -mosques -relevance -##wine -wta -##23 -##vah -##lter -hoc -##riding -optimistic -##´s -deco -sim -interacting -rejecting -moniker -waterways -##ieri -##oku -mayors -gdansk -outnumbered -pearls -##ended -##hampton -fairs -totals -dominating -262 -notions -stairway -compiling -pursed -commodities -grease -yeast -##jong -carthage -griffiths -residual -amc -contraction -laird -sapphire -##marine -##ivated -amalgamation -dissolve -inclination -lyle -packaged -altitudes -suez -canons -graded -lurched -narrowing -boasts -guise -wed -enrico -##ovsky -rower -scarred -bree -cub -iberian -protagonists -bargaining -proposing -trainers -voyages -vans -fishes -##aea -##ivist -##verance -encryption -artworks -kazan -sabre -cleopatra -hepburn -rotting -supremacy -mecklenburg -##brate -burrows -hazards -outgoing -flair -organizes -##ctions -scorpion -##usions -boo -234 -chevalier -dunedin -slapping -##34 -ineligible -pensions -##38 -##omic -manufactures -emails -bismarck -238 -weakening -blackish -ding -mcgee -quo -##rling -northernmost -xx -manpower -greed -sampson -clicking -##ange -##horpe -##inations -##roving -torre -##eptive -##moral -symbolism -38th -asshole -meritorious -outfits -splashed -biographies -sprung -astros -##tale -302 -737 -filly -raoul -nw -tokugawa -linden -clubhouse -##apa -tracts -romano -##pio -putin -tags -##note -chained -dickson -gunshot -moe -gunn -rashid -##tails -zipper -##bas -##nea -contrasted -##ply -##udes -plum -pharaoh -##pile -aw -comedies -ingrid -sandwiches -subdivisions -1100 -mariana -nokia -kamen -hz -delaney -veto -herring -##words -possessive -outlines -##roup -siemens -stairwell -rc -gallantry -messiah -palais -yells -233 -zeppelin -##dm -bolivar -##cede -smackdown -mckinley -##mora -##yt -muted -geologic -finely -unitary -avatar -hamas -maynard -rees -bog -contrasting -##rut -liv -chico -disposition -pixel -##erate -becca -dmitry -yeshiva -narratives -##lva -##ulton -mercenary -sharpe -tempered -navigate -stealth -amassed -keynes -##lini -untouched -##rrie -havoc -lithium -##fighting -abyss -graf -southward -wolverine -balloons -implements -ngos -transitions -##icum -ambushed -concacaf -dormant -economists -##dim -costing -csi -rana -universite -boulders -verity -##llon -collin -mellon -misses -cypress -fluorescent -lifeless -spence -##ulla -crewe -shepard -pak -revelations -##م -jolly -gibbons -paw -##dro -##quel -freeing -##test -shack -fries -palatine -##51 -##hiko -accompaniment -cruising -recycled -##aver -erwin -sorting -synthesizers -dyke -realities -sg -strides -enslaved -wetland -##ghan -competence -gunpowder -grassy -maroon -reactors -objection -##oms -carlson -gearbox -macintosh -radios -shelton -##sho -clergyman -prakash -254 -mongols -trophies -oricon -228 -stimuli -twenty20 -cantonese -cortes -mirrored -##saurus -bhp -cristina -melancholy -##lating -enjoyable -nuevo -##wny -downfall -schumacher -##ind -banging -lausanne -rumbled -paramilitary -reflex -ax -amplitude -migratory -##gall -##ups -midi -barnard -lastly -sherry -##hp -##nall -keystone -##kra -carleton -slippery -##53 -coloring -foe -socket -otter -##rgos -mats -##tose -consultants -bafta -bison -topping -##km -490 -primal -abandonment -transplant -atoll -hideous -mort -pained -reproduced -tae -howling -##turn -unlawful -billionaire -hotter -poised -lansing -##chang -dinamo -retro -messing -nfc -domesday -##mina -blitz -timed -##athing -##kley -ascending -gesturing -##izations -signaled -tis -chinatown -mermaid -savanna -jameson -##aint -catalina -##pet -##hers -cochrane -cy -chatting -##kus -alerted -computation -mused -noelle -majestic -mohawk -campo -octagonal -##sant -##hend -241 -aspiring -##mart -comprehend -iona -paralyzed -shimmering -swindon -rhone -##eley -reputed -configurations -pitchfork -agitation -francais -gillian -lipstick -##ilo -outsiders -pontifical -resisting -bitterness -sewer -rockies -##edd -##ucher -misleading -1756 -exiting -galloway -##nging -risked -##heart -246 -commemoration -schultz -##rka -integrating -##rsa -poses -shrieked -##weiler -guineas -gladys -jerking -owls -goldsmith -nightly -penetrating -##unced -lia -##33 -ignited -betsy -##aring -##thorpe -follower -vigorously -##rave -coded -kiran -knit -zoology -tbilisi -##28 -##bered -repository -govt -deciduous -dino -growling -##bba -enhancement -unleashed -chanting -pussy -biochemistry -##eric -kettle -repression -toxicity -nrhp -##arth -##kko -##bush -ernesto -commended -outspoken -242 -mca -parchment -sms -kristen -##aton -bisexual -raked -glamour -navajo -a2 -conditioned -showcased -##hma -spacious -youthful -##esa -usl -appliances -junta -brest -layne -conglomerate -enchanted -chao -loosened -picasso -circulating -inspect -montevideo -##centric -##kti -piazza -spurred -##aith -bari -freedoms -poultry -stamford -lieu -##ect -indigo -sarcastic -bahia -stump -attach -dvds -frankenstein -lille -approx -scriptures -pollen -##script -nmi -overseen -##ivism -tides -proponent -newmarket -inherit -milling -##erland -centralized -##rou -distributors -credentials -drawers -abbreviation -##lco -##xon -downing -uncomfortably -ripe -##oes -erase -franchises -##ever -populace -##bery -##khar -decomposition -pleas -##tet -daryl -sabah -##stle -##wide -fearless -genie -lesions -annette -##ogist -oboe -appendix -nair -dripped -petitioned -maclean -mosquito -parrot -rpg -hampered -1648 -operatic -reservoirs -##tham -irrelevant -jolt -summarized -##fp -medallion -##taff -##− -clawed -harlow -narrower -goddard -marcia -bodied -fremont -suarez -altering -tempest -mussolini -porn -##isms -sweetly -oversees -walkers -solitude -grimly -shrines -hk -ich -supervisors -hostess -dietrich -legitimacy -brushes -expressive -##yp -dissipated -##rse -localized -systemic -##nikov -gettysburg -##js -##uaries -dialogues -muttering -251 -housekeeper -sicilian -discouraged -##frey -beamed -kaladin -halftime -kidnap -##amo -##llet -1754 -synonymous -depleted -instituto -insulin -reprised -##opsis -clashed -##ctric -interrupting -radcliffe -insisting -medici -1715 -ejected -playfully -turbulent -##47 -starvation -##rini -shipment -rebellious -petersen -verification -merits -##rified -cakes -##charged -1757 -milford -shortages -spying -fidelity -##aker -emitted -storylines -harvested -seismic -##iform -cheung -kilda -theoretically -barbie -lynx -##rgy -##tius -goblin -mata -poisonous -##nburg -reactive -residues -obedience -##евич -conjecture -##rac -401 -hating -sixties -kicker -moaning -motown -##bha -emancipation -neoclassical -##hering -consoles -ebert -professorship -##tures -sustaining -assaults -obeyed -affluent -incurred -tornadoes -##eber -##zow -emphasizing -highlanders -cheated -helmets -##ctus -internship -terence -bony -executions -legislators -berries -peninsular -tinged -##aco -1689 -amplifier -corvette -ribbons -lavish -pennant -##lander -worthless -##chfield -##forms -mariano -pyrenees -expenditures -##icides -chesterfield -mandir -tailor -39th -sergey -nestled -willed -aristocracy -devotees -goodnight -raaf -rumored -weaponry -remy -appropriations -harcourt -burr -riaa -##lence -limitation -unnoticed -guo -soaking -swamps -##tica -collapsing -tatiana -descriptive -brigham -psalm -##chment -maddox -##lization -patti -caliph -##aja -akron -injuring -serra -##ganj -basins -##sari -astonished -launcher -##church -hilary -wilkins -sewing -##sf -stinging -##fia -##ncia -underwood -startup -##ition -compilations -vibrations -embankment -jurist -##nity -bard -juventus -groundwater -kern -palaces -helium -boca -cramped -marissa -soto -##worm -jae -princely -##ggy -faso -bazaar -warmly -##voking -229 -pairing -##lite -##grate -##nets -wien -freaked -ulysses -rebirth -##alia -##rent -mummy -guzman -jimenez -stilled -##nitz -trajectory -tha -woken -archival -professions -##pts -##pta -hilly -shadowy -shrink -##bolt -norwood -glued -migrate -stereotypes -devoid -##pheus -625 -evacuate -horrors -infancy -gotham -knowles -optic -downloaded -sachs -kingsley -parramatta -darryl -mor -##onale -shady -commence -confesses -kan -##meter -##placed -marlborough -roundabout -regents -frigates -io -##imating -gothenburg -revoked -carvings -clockwise -convertible -intruder -##sche -banged -##ogo -vicky -bourgeois -##mony -dupont -footing -##gum -pd -##real -buckle -yun -penthouse -sane -720 -serviced -stakeholders -neumann -bb -##eers -comb -##gam -catchment -pinning -rallies -typing -##elles -forefront -freiburg -sweetie -giacomo -widowed -goodwill -worshipped -aspirations -midday -##vat -fishery -##trick -bournemouth -turk -243 -hearth -ethanol -guadalajara -murmurs -sl -##uge -afforded -scripted -##hta -wah -##jn -coroner -translucent -252 -memorials -puck -progresses -clumsy -##race -315 -candace -recounted -##27 -##slin -##uve -filtering -##mac -howl -strata -heron -leveled -##ays -dubious -##oja -##т -##wheel -citations -exhibiting -##laya -##mics -##pods -turkic -##lberg -injunction -##ennial -##mit -antibodies -##44 -organise -##rigues -cardiovascular -cushion -inverness -##zquez -dia -cocoa -sibling -##tman -##roid -expanse -feasible -tunisian -algiers -##relli -rus -bloomberg -dso -westphalia -bro -tacoma -281 -downloads -##ours -konrad -duran -##hdi -continuum -jett -compares -legislator -secession -##nable -##gues -##zuka -translating -reacher -##gley -##ła -aleppo -##agi -tc -orchards -trapping -linguist -versatile -drumming -postage -calhoun -superiors -##mx -barefoot -leary -##cis -ignacio -alfa -kaplan -##rogen -bratislava -mori -##vot -disturb -haas -313 -cartridges -gilmore -radiated -salford -tunic -hades -##ulsive -archeological -delilah -magistrates -auditioned -brewster -charters -empowerment -blogs -cappella -dynasties -iroquois -whipping -##krishna -raceway -truths -myra -weaken -judah -mcgregor -##horse -mic -refueling -37th -burnley -bosses -markus -premio -query -##gga -dunbar -##economic -darkest -lyndon -sealing -commendation -reappeared -##mun -addicted -ezio -slaughtered -satisfactory -shuffle -##eves -##thic -##uj -fortification -warrington -##otto -resurrected -fargo -mane -##utable -##lei -##space -foreword -ox -##aris -##vern -abrams -hua -##mento -sakura -##alo -uv -sentimental -##skaya -midfield -##eses -sturdy -scrolls -macleod -##kyu -entropy -##lance -mitochondrial -cicero -excelled -thinner -convoys -perceive -##oslav -##urable -systematically -grind -burkina -287 -##tagram -ops -##aman -guantanamo -##cloth -##tite -forcefully -wavy -##jou -pointless -##linger -##tze -layton -portico -superficial -clerical -outlaws -##hism -burials -muir -##inn -creditors -hauling -rattle -##leg -calais -monde -archers -reclaimed -dwell -wexford -hellenic -falsely -remorse -##tek -dough -furnishings -##uttered -gabon -neurological -novice -##igraphy -contemplated -pulpit -nightstand -saratoga -##istan -documenting -pulsing -taluk -##firmed -busted -marital -##rien -disagreements -wasps -##yes -hodge -mcdonnell -mimic -fran -pendant -dhabi -musa -##nington -congratulations -argent -darrell -concussion -losers -regrets -thessaloniki -reversal -donaldson -hardwood -thence -achilles -ritter -##eran -demonic -jurgen -prophets -goethe -eki -classmate -buff -##cking -yank -irrational -##inging -perished -seductive -qur -sourced -##crat -##typic -mustard -ravine -barre -horizontally -characterization -phylogenetic -boise -##dit -##runner -##tower -brutally -intercourse -seduce -##bbing -fay -ferris -ogden -amar -nik -unarmed -##inator -evaluating -kyrgyzstan -sweetness -##lford -##oki -mccormick -meiji -notoriety -stimulate -disrupt -figuring -instructional -mcgrath -##zoo -groundbreaking -##lto -flinch -khorasan -agrarian -bengals -mixer -radiating -##sov -ingram -pitchers -nad -tariff -##cript -tata -##codes -##emi -##ungen -appellate -lehigh -##bled -##giri -brawl -duct -texans -##ciation -##ropolis -skipper -speculative -vomit -doctrines -stresses -253 -davy -graders -whitehead -jozef -timely -cumulative -haryana -paints -appropriately -boon -cactus -##ales -##pid -dow -legions -##pit -perceptions -1730 -picturesque -##yse -periphery -rune -wr -##aha -celtics -sentencing -whoa -##erin -confirms -variance -425 -moines -mathews -spade -rave -m1 -fronted -fx -blending -alleging -reared -##gl -237 -##paper -grassroots -eroded -##free -##physical -directs -ordeal -##sław -accelerate -hacker -rooftop -##inia -lev -buys -cebu -devote -##lce -specialising -##ulsion -choreographed -repetition -warehouses -##ryl -paisley -tuscany -analogy -sorcerer -hash -huts -shards -descends -exclude -nix -chaplin -gaga -ito -vane -##drich -causeway -misconduct -limo -orchestrated -glands -jana -##kot -u2 -##mple -##sons -branching -contrasts -scoop -longed -##virus -chattanooga -##75 -syrup -cornerstone -##tized -##mind -##iaceae -careless -precedence -frescoes -##uet -chilled -consult -modelled -snatch -peat -##thermal -caucasian -humane -relaxation -spins -temperance -##lbert -occupations -lambda -hybrids -moons -mp3 -##oese -247 -rolf -societal -yerevan -ness -##ssler -befriended -mechanized -nominate -trough -boasted -cues -seater -##hom -bends -##tangle -conductors -emptiness -##lmer -eurasian -adriatic -tian -##cie -anxiously -lark -propellers -chichester -jock -ev -2a -##holding -credible -recounts -tori -loyalist -abduction -##hoot -##redo -nepali -##mite -ventral -tempting -##ango -##crats -steered -##wice -javelin -dipping -laborers -prentice -looming -titanium -##ː -badges -emir -tensor -##ntation -egyptians -rash -denies -hawthorne -lombard -showers -wehrmacht -dietary -trojan -##reus -welles -executing -horseshoe -lifeboat -##lak -elsa -infirmary -nearing -roberta -boyer -mutter -trillion -joanne -##fine -##oked -sinks -vortex -uruguayan -clasp -sirius -##block -accelerator -prohibit -sunken -byu -chronological -diplomats -ochreous -510 -symmetrical -1644 -maia -##tology -salts -reigns -atrocities -##ия -hess -bared -issn -##vyn -cater -saturated -##cycle -##isse -sable -voyager -dyer -yusuf -##inge -fountains -wolff -##39 -##nni -engraving -rollins -atheist -ominous -##ault -herr -chariot -martina -strung -##fell -##farlane -horrific -sahib -gazes -saetan -erased -ptolemy -##olic -flushing -lauderdale -analytic -##ices -530 -navarro -beak -gorilla -herrera -broom -guadalupe -raiding -sykes -311 -bsc -deliveries -1720 -invasions -carmichael -tajikistan -thematic -ecumenical -sentiments -onstage -##rians -##brand -##sume -catastrophic -flanks -molten -##arns -waller -aimee -terminating -##icing -alternately -##oche -nehru -printers -outraged -##eving -empires -template -banners -repetitive -za -##oise -vegetarian -##tell -guiana -opt -cavendish -lucknow -synthesized -##hani -##mada -finalized -##ctable -fictitious -mayoral -unreliable -##enham -embracing -peppers -rbis -##chio -##neo -inhibition -slashed -togo -orderly -embroidered -safari -salty -236 -barron -benito -totaled -##dak -pubs -simulated -caden -devin -tolkien -momma -welding -sesame -##ept -gottingen -hardness -630 -shaman -temeraire -620 -adequately -pediatric -##kit -ck -assertion -radicals -composure -cadence -seafood -beaufort -lazarus -mani -warily -cunning -kurdistan -249 -cantata -##kir -ares -##41 -##clusive -nape -townland -geared -insulted -flutter -boating -violate -draper -dumping -malmo -##hh -##romatic -firearm -alta -bono -obscured -##clave -exceeds -panorama -unbelievable -##train -preschool -##essed -disconnected -installing -rescuing -secretaries -accessibility -##castle -##drive -##ifice -##film -bouts -slug -waterway -mindanao -##buro -##ratic -halves -##ل -calming -liter -maternity -adorable -bragg -electrification -mcc -##dote -roxy -schizophrenia -##body -munoz -kaye -whaling -239 -mil -tingling -tolerant -##ago -unconventional -volcanoes -##finder -deportivo -##llie -robson -kaufman -neuroscience -wai -deportation -masovian -scraping -converse -##bh -hacking -bulge -##oun -administratively -yao -580 -amp -mammoth -booster -claremont -hooper -nomenclature -pursuits -mclaughlin -melinda -##sul -catfish -barclay -substrates -taxa -zee -originals -kimberly -packets -padma -##ality -borrowing -ostensibly -solvent -##bri -##genesis -##mist -lukas -shreveport -veracruz -##ь -##lou -##wives -cheney -tt -anatolia -hobbs -##zyn -cyclic -radiant -alistair -greenish -siena -dat -independents -##bation -conform -pieter -hyper -applicant -bradshaw -spores -telangana -vinci -inexpensive -nuclei -322 -jang -nme -soho -spd -##ign -cradled -receptionist -pow -##43 -##rika -fascism -##ifer -experimenting -##ading -##iec -##region -345 -jocelyn -maris -stair -nocturnal -toro -constabulary -elgin -##kker -msc -##giving -##schen -##rase -doherty -doping -sarcastically -batter -maneuvers -##cano -##apple -##gai -##git -intrinsic -##nst -##stor -1753 -showtime -cafes -gasps -lviv -ushered -##thed -fours -restart -astonishment -transmitting -flyer -shrugs -##sau -intriguing -cones -dictated -mushrooms -medial -##kovsky -##elman -escorting -gaped -##26 -godfather -##door -##sell -djs -recaptured -timetable -vila -1710 -3a -aerodrome -mortals -scientology -##orne -angelina -mag -convection -unpaid -insertion -intermittent -lego -##nated -endeavor -kota -pereira -##lz -304 -bwv -glamorgan -insults -agatha -fey -##cend -fleetwood -mahogany -protruding -steamship -zeta -##arty -mcguire -suspense -##sphere -advising -urges -##wala -hurriedly -meteor -gilded -inline -arroyo -stalker -##oge -excitedly -revered -##cure -earle -introductory -##break -##ilde -mutants -puff -pulses -reinforcement -##haling -curses -lizards -stalk -correlated -##fixed -fallout -macquarie -##unas -bearded -denton -heaving -802 -##ocation -winery -assign -dortmund -##lkirk -everest -invariant -charismatic -susie -##elling -bled -lesley -telegram -sumner -bk -##ogen -##к -wilcox -needy -colbert -duval -##iferous -##mbled -allotted -attends -imperative -##hita -replacements -hawker -##inda -insurgency -##zee -##eke -casts -##yla -680 -ives -transitioned -##pack -##powering -authoritative -baylor -flex -cringed -plaintiffs -woodrow -##skie -drastic -ape -aroma -unfolded -commotion -nt -preoccupied -theta -routines -lasers -privatization -wand -domino -ek -clenching -nsa -strategically -showered -bile -handkerchief -pere -storing -christophe -insulting -316 -nakamura -romani -asiatic -magdalena -palma -cruises -stripping -405 -konstantin -soaring -##berman -colloquially -forerunner -havilland -incarcerated -parasites -sincerity -##utus -disks -plank -saigon -##ining -corbin -homo -ornaments -powerhouse -##tlement -chong -fastened -feasibility -idf -morphological -usable -##nish -##zuki -aqueduct -jaguars -keepers -##flies -aleksandr -faust -assigns -ewing -bacterium -hurled -tricky -hungarians -integers -wallis -321 -yamaha -##isha -hushed -oblivion -aviator -evangelist -friars -##eller -monograph -ode -##nary -airplanes -labourers -charms -##nee -1661 -hagen -tnt -rudder -fiesta -transcript -dorothea -ska -inhibitor -maccabi -retorted -raining -encompassed -clauses -menacing -1642 -lineman -##gist -vamps -##ape -##dick -gloom -##rera -dealings -easing -seekers -##nut -##pment -helens -unmanned -##anu -##isson -basics -##amy -##ckman -adjustments -1688 -brutality -horne -##zell -sui -##55 -##mable -aggregator -##thal -rhino -##drick -##vira -counters -zoom -##01 -##rting -mn -montenegrin -packard -##unciation -##♭ -##kki -reclaim -scholastic -thugs -pulsed -##icia -syriac -quan -saddam -banda -kobe -blaming -buddies -dissent -##lusion -##usia -corbett -jaya -delle -erratic -lexie -##hesis -435 -amiga -hermes -##pressing -##leen -chapels -gospels -jamal -##uating -compute -revolving -warp -##sso -##thes -armory -##eras -##gol -antrim -loki -##kow -##asian -##good -##zano -braid -handwriting -subdistrict -funky -pantheon -##iculate -concurrency -estimation -improper -juliana -##his -newcomers -johnstone -staten -communicated -##oco -##alle -sausage -stormy -##stered -##tters -superfamily -##grade -acidic -collateral -tabloid -##oped -##rza -bladder -austen -##ellant -mcgraw -##hay -hannibal -mein -aquino -lucifer -wo -badger -boar -cher -christensen -greenberg -interruption -##kken -jem -244 -mocked -bottoms -cambridgeshire -##lide -sprawling -##bbly -eastwood -ghent -synth -##buck -advisers -##bah -nominally -hapoel -qu -daggers -estranged -fabricated -towels -vinnie -wcw -misunderstanding -anglia -nothin -unmistakable -##dust -##lova -chilly -marquette -truss -##edge -##erine -reece -##lty -##chemist -##connected -272 -308 -41st -bash -raion -waterfalls -##ump -##main -labyrinth -queue -theorist -##istle -bharatiya -flexed -soundtracks -rooney -leftist -patrolling -wharton -plainly -alleviate -eastman -schuster -topographic -engages -immensely -unbearable -fairchild -1620 -dona -lurking -parisian -oliveira -ia -indictment -hahn -bangladeshi -##aster -vivo -##uming -##ential -antonia -expects -indoors -kildare -harlan -##logue -##ogenic -##sities -forgiven -##wat -childish -tavi -##mide -##orra -plausible -grimm -successively -scooted -##bola -##dget -##rith -spartans -emery -flatly -azure -epilogue -##wark -flourish -##iny -##tracted -##overs -##oshi -bestseller -distressed -receipt -spitting -hermit -topological -##cot -drilled -subunit -francs -##layer -eel -##fk -##itas -octopus -footprint -petitions -ufo -##say -##foil -interfering -leaking -palo -##metry -thistle -valiant -##pic -narayan -mcpherson -##fast -gonzales -##ym -##enne -dustin -novgorod -solos -##zman -doin -##raph -##patient -##meyer -soluble -ashland -cuffs -carole -pendleton -whistling -vassal -##river -deviation -revisited -constituents -rallied -rotate -loomed -##eil -##nting -amateurs -augsburg -auschwitz -crowns -skeletons -##cona -bonnet -257 -dummy -globalization -simeon -sleeper -mandal -differentiated -##crow -##mare -milne -bundled -exasperated -talmud -owes -segregated -##feng -##uary -dentist -piracy -props -##rang -devlin -##torium -malicious -paws -##laid -dependency -##ergy -##fers -##enna -258 -pistons -rourke -jed -grammatical -tres -maha -wig -512 -ghostly -jayne -##achal -##creen -##ilis -##lins -##rence -designate -##with -arrogance -cambodian -clones -showdown -throttle -twain -##ception -lobes -metz -nagoya -335 -braking -##furt -385 -roaming -##minster -amin -crippled -##37 -##llary -indifferent -hoffmann -idols -intimidating -1751 -261 -influenza -memo -onions -1748 -bandage -consciously -##landa -##rage -clandestine -observes -swiped -tangle -##ener -##jected -##trum -##bill -##lta -hugs -congresses -josiah -spirited -##dek -humanist -managerial -filmmaking -inmate -rhymes -debuting -grimsby -ur -##laze -duplicate -vigor -##tf -republished -bolshevik -refurbishment -antibiotics -martini -methane -newscasts -royale -horizons -levant -iain -visas -##ischen -paler -##around -manifestation -snuck -alf -chop -futile -pedestal -rehab -##kat -bmg -kerman -res -fairbanks -jarrett -abstraction -saharan -##zek -1746 -procedural -clearer -kincaid -sash -luciano -##ffey -crunch -helmut -##vara -revolutionaries -##tute -creamy -leach -##mmon -1747 -permitting -nes -plight -wendell -##lese -contra -ts -clancy -ipa -mach -staples -autopsy -disturbances -nueva -karin -pontiac -##uding -proxy -venerable -haunt -leto -bergman -expands -##helm -wal -##pipe -canning -celine -cords -obesity -##enary -intrusion -planner -##phate -reasoned -sequencing -307 -harrow -##chon -##dora -marred -mcintyre -repay -tarzan -darting -248 -harrisburg -margarita -repulsed -##hur -##lding -belinda -hamburger -novo -compliant -runways -bingham -registrar -skyscraper -ic -cuthbert -improvisation -livelihood -##corp -##elial -admiring -##dened -sporadic -believer -casablanca -popcorn -##29 -asha -shovel -##bek -##dice -coiled -tangible -##dez -casper -elsie -resin -tenderness -rectory -##ivision -avail -sonar -##mori -boutique -##dier -guerre -bathed -upbringing -vaulted -sandals -blessings -##naut -##utnant -1680 -306 -foxes -pia -corrosion -hesitantly -confederates -crystalline -footprints -shapiro -tirana -valentin -drones -45th -microscope -shipments -texted -inquisition -wry -guernsey -unauthorized -resigning -760 -ripple -schubert -stu -reassure -felony -##ardo -brittle -koreans -##havan -##ives -dun -implicit -tyres -##aldi -##lth -magnolia -##ehan -##puri -##poulos -aggressively -fei -gr -familiarity -##poo -indicative -##trust -fundamentally -jimmie -overrun -395 -anchors -moans -##opus -britannia -armagh -##ggle -purposely -seizing -##vao -bewildered -mundane -avoidance -cosmopolitan -geometridae -quartermaster -caf -415 -chatter -engulfed -gleam -purge -##icate -juliette -jurisprudence -guerra -revisions -##bn -casimir -brew -##jm -1749 -clapton -cloudy -conde -hermitage -278 -simulations -torches -vincenzo -matteo -##rill -hidalgo -booming -westbound -accomplishment -tentacles -unaffected -##sius -annabelle -flopped -sloping -##litz -dreamer -interceptor -vu -##loh -consecration -copying -messaging -breaker -climates -hospitalized -1752 -torino -afternoons -winfield -witnessing -##teacher -breakers -choirs -sawmill -coldly -##ege -sipping -haste -uninhabited -conical -bibliography -pamphlets -severn -edict -##oca -deux -illnesses -grips -##pl -rehearsals -sis -thinkers -tame -##keepers -1690 -acacia -reformer -##osed -##rys -shuffling -##iring -##shima -eastbound -ionic -rhea -flees -littered -##oum -rocker -vomiting -groaning -champ -overwhelmingly -civilizations -paces -sloop -adoptive -##tish -skaters -##vres -aiding -mango -##joy -nikola -shriek -##ignon -pharmaceuticals -##mg -tuna -calvert -gustavo -stocked -yearbook -##urai -##mana -computed -subsp -riff -hanoi -kelvin -hamid -moors -pastures -summons -jihad -nectar -##ctors -bayou -untitled -pleasing -vastly -republics -intellect -##η -##ulio -##tou -crumbling -stylistic -sb -##ی -consolation -frequented -h₂o -walden -widows -##iens -404 -##ignment -chunks -improves -288 -grit -recited -##dev -snarl -sociological -##arte -##gul -inquired -##held -bruise -clube -consultancy -homogeneous -hornets -multiplication -pasta -prick -savior -##grin -##kou -##phile -yoon -##gara -grimes -vanishing -cheering -reacting -bn -distillery -##quisite -##vity -coe -dockyard -massif -##jord -escorts -voss -##valent -byte -chopped -hawke -illusions -workings -floats -##koto -##vac -kv -annapolis -madden -##onus -alvaro -noctuidae -##cum -##scopic -avenge -steamboat -forte -illustrates -erika -##trip -570 -dew -nationalities -bran -manifested -thirsty -diversified -muscled -reborn -##standing -arson -##lessness -##dran -##logram -##boys -##kushima -##vious -willoughby -##phobia -286 -alsace -dashboard -yuki -##chai -granville -myspace -publicized -tricked -##gang -adjective -##ater -relic -reorganisation -enthusiastically -indications -saxe -##lassified -consolidate -iec -padua -helplessly -ramps -renaming -regulars -pedestrians -accents -convicts -inaccurate -lowers -mana -##pati -barrie -bjp -outta -someplace -berwick -flanking -invoked -marrow -sparsely -excerpts -clothed -rei -##ginal -wept -##straße -##vish -alexa -excel -##ptive -membranes -aquitaine -creeks -cutler -sheppard -implementations -ns -##dur -fragrance -budge -concordia -magnesium -marcelo -##antes -gladly -vibrating -##rral -##ggles -montrose -##omba -lew -seamus -1630 -cocky -##ament -##uen -bjorn -##rrick -fielder -fluttering -##lase -methyl -kimberley -mcdowell -reductions -barbed -##jic -##tonic -aeronautical -condensed -distracting -##promising -huffed -##cala -##sle -claudius -invincible -missy -pious -balthazar -ci -##lang -butte -combo -orson -##dication -myriad -1707 -silenced -##fed -##rh -coco -netball -yourselves -##oza -clarify -heller -peg -durban -etudes -offender -roast -blackmail -curvature -##woods -vile -309 -illicit -suriname -##linson -overture -1685 -bubbling -gymnast -tucking -##mming -##ouin -maldives -##bala -gurney -##dda -##eased -##oides -backside -pinto -jars -racehorse -tending -##rdial -baronetcy -wiener -duly -##rke -barbarian -cupping -flawed -##thesis -bertha -pleistocene -puddle -swearing -##nob -##tically -fleeting -prostate -amulet -educating -##mined -##iti -##tler -75th -jens -respondents -analytics -cavaliers -papacy -raju -##iente -##ulum -##tip -funnel -271 -disneyland -##lley -sociologist -##iam -2500 -faulkner -louvre -menon -##dson -276 -##ower -afterlife -mannheim -peptide -referees -comedians -meaningless -##anger -##laise -fabrics -hurley -renal -sleeps -##bour -##icle -breakout -kristin -roadside -animator -clover -disdain -unsafe -redesign -##urity -firth -barnsley -portage -reset -narrows -268 -commandos -expansive -speechless -tubular -##lux -essendon -eyelashes -smashwords -##yad -##bang -##claim -craved -sprinted -chet -somme -astor -wrocław -orton -266 -bane -##erving -##uing -mischief -##amps -##sund -scaling -terre -##xious -impairment -offenses -undermine -moi -soy -contiguous -arcadia -inuit -seam -##tops -macbeth -rebelled -##icative -##iot -590 -elaborated -frs -uniformed -##dberg -259 -powerless -priscilla -stimulated -980 -qc -arboretum -frustrating -trieste -bullock -##nified -enriched -glistening -intern -##adia -locus -nouvelle -ollie -ike -lash -starboard -ee -tapestry -headlined -hove -rigged -##vite -pollock -##yme -thrive -clustered -cas -roi -gleamed -olympiad -##lino -pressured -regimes -##hosis -##lick -ripley -##ophone -kickoff -gallon -rockwell -##arable -crusader -glue -revolutions -scrambling -1714 -grover -##jure -englishman -aztec -263 -contemplating -coven -ipad -preach -triumphant -tufts -##esian -rotational -##phus -328 -falkland -##brates -strewn -clarissa -rejoin -environmentally -glint -banded -drenched -moat -albanians -johor -rr -maestro -malley -nouveau -shaded -taxonomy -v6 -adhere -bunk -airfields -##ritan -1741 -encompass -remington -tran -##erative -amelie -mazda -friar -morals -passions -##zai -breadth -vis -##hae -argus -burnham -caressing -insider -rudd -##imov -##mini -##rso -italianate -murderous -textual -wainwright -armada -bam -weave -timer -##taken -##nh -fra -##crest -ardent -salazar -taps -tunis -##ntino -allegro -gland -philanthropic -##chester -implication -##optera -esq -judas -noticeably -wynn -##dara -inched -indexed -crises -villiers -bandit -royalties -patterned -cupboard -interspersed -accessory -isla -kendrick -entourage -stitches -##esthesia -headwaters -##ior -interlude -distraught -draught -1727 -##basket -biased -sy -transient -triad -subgenus -adapting -kidd -shortstop -##umatic -dimly -spiked -mcleod -reprint -nellie -pretoria -windmill -##cek -singled -##mps -273 -reunite -##orous -747 -bankers -outlying -##omp -##ports -##tream -apologies -cosmetics -patsy -##deh -##ocks -##yson -bender -nantes -serene -##nad -lucha -mmm -323 -##cius -##gli -cmll -coinage -nestor -juarez -##rook -smeared -sprayed -twitching -sterile -irina -embodied -juveniles -enveloped -miscellaneous -cancers -dq -gulped -luisa -crested -swat -donegal -ref -##anov -##acker -hearst -mercantile -##lika -doorbell -ua -vicki -##alla -##som -bilbao -psychologists -stryker -sw -horsemen -turkmenistan -wits -##national -anson -mathew -screenings -##umb -rihanna -##agne -##nessy -aisles -##iani -##osphere -hines -kenton -saskatoon -tasha -truncated -##champ -##itan -mildred -advises -fredrik -interpreting -inhibitors -##athi -spectroscopy -##hab -##kong -karim -panda -##oia -##nail -##vc -conqueror -kgb -leukemia -##dity -arrivals -cheered -pisa -phosphorus -shielded -##riated -mammal -unitarian -urgently -chopin -sanitary -##mission -spicy -drugged -hinges -##tort -tipping -trier -impoverished -westchester -##caster -267 -epoch -nonstop -##gman -##khov -aromatic -centrally -cerro -##tively -##vio -billions -modulation -sedimentary -283 -facilitating -outrageous -goldstein -##eak -##kt -ld -maitland -penultimate -pollard -##dance -fleets -spaceship -vertebrae -##nig -alcoholism -als -recital -##bham -##ference -##omics -m2 -##bm -trois -##tropical -##в -commemorates -##meric -marge -##raction -1643 -670 -cosmetic -ravaged -##ige -catastrophe -eng -##shida -albrecht -arterial -bellamy -decor -harmon -##rde -bulbs -synchronized -vito -easiest -shetland -shielding -wnba -##glers -##ssar -##riam -brianna -cumbria -##aceous -##rard -cores -thayer -##nsk -brood -hilltop -luminous -carts -keynote -larkin -logos -##cta -##ا -##mund -##quay -lilith -tinted -277 -wrestle -mobilization -##uses -sequential -siam -bloomfield -takahashi -274 -##ieving -presenters -ringo -blazed -witty -##oven -##ignant -devastation -haydn -harmed -newt -therese -##peed -gershwin -molina -rabbis -sudanese -001 -innate -restarted -##sack -##fus -slices -wb -##shah -enroll -hypothetical -hysterical -1743 -fabio -indefinite -warped -##hg -exchanging -525 -unsuitable -##sboro -gallo -1603 -bret -cobalt -homemade -##hunter -mx -operatives -##dhar -terraces -durable -latch -pens -whorls -##ctuated -##eaux -billing -ligament -succumbed -##gly -regulators -spawn -##brick -##stead -filmfare -rochelle -##nzo -1725 -circumstance -saber -supplements -##nsky -##tson -crowe -wellesley -carrot -##9th -##movable -primate -drury -sincerely -topical -##mad -##rao -callahan -kyiv -smarter -tits -undo -##yeh -announcements -anthologies -barrio -nebula -##islaus -##shaft -##tyn -bodyguards -2021 -assassinate -barns -emmett -scully -##mah -##yd -##eland -##tino -##itarian -demoted -gorman -lashed -prized -adventist -writ -##gui -alla -invertebrates -##ausen -1641 -amman -1742 -align -healy -redistribution -##gf -##rize -insulation -##drop -adherents -hezbollah -vitro -ferns -yanking -269 -php -registering -uppsala -cheerleading -confines -mischievous -tully -##ross -49th -docked -roam -stipulated -pumpkin -##bry -prompt -##ezer -blindly -shuddering -craftsmen -frail -scented -katharine -scramble -shaggy -sponge -helix -zaragoza -279 -##52 -43rd -backlash -fontaine -seizures -posse -cowan -nonfiction -telenovela -wwii -hammered -undone -##gpur -encircled -irs -##ivation -artefacts -oneself -searing -smallpox -##belle -##osaurus -shandong -breached -upland -blushing -rankin -infinitely -psyche -tolerated -docking -evicted -##col -unmarked -##lving -gnome -lettering -litres -musique -##oint -benevolent -##jal -blackened -##anna -mccall -racers -tingle -##ocene -##orestation -introductions -radically -292 -##hiff -##باد -1610 -1739 -munchen -plead -##nka -condo -scissors -##sight -##tens -apprehension -##cey -##yin -hallmark -watering -formulas -sequels -##llas -aggravated -bae -commencing -##building -enfield -prohibits -marne -vedic -civilized -euclidean -jagger -beforehand -blasts -dumont -##arney -##nem -740 -conversions -hierarchical -rios -simulator -##dya -##lellan -hedges -oleg -thrusts -shadowed -darby -maximize -1744 -gregorian -##nded -##routed -sham -unspecified -##hog -emory -factual -##smo -##tp -fooled -##rger -ortega -wellness -marlon -##oton -##urance -casket -keating -ley -enclave -##ayan -char -influencing -jia -##chenko -412 -ammonia -erebidae -incompatible -violins -cornered -##arat -grooves -astronauts -columbian -rampant -fabrication -kyushu -mahmud -vanish -##dern -mesopotamia -##lete -ict -##rgen -caspian -kenji -pitted -##vered -999 -grimace -roanoke -tchaikovsky -twinned -##analysis -##awan -xinjiang -arias -clemson -kazakh -sizable -1662 -##khand -##vard -plunge -tatum -vittorio -##nden -cholera -##dana -##oper -bracing -indifference -projectile -superliga -##chee -realises -upgrading -299 -porte -retribution -##vies -nk -stil -##resses -ama -bureaucracy -blackberry -bosch -testosterone -collapses -greer -##pathic -ioc -fifties -malls -##erved -bao -baskets -adolescents -siegfried -##osity -##tosis -mantra -detecting -existent -fledgling -##cchi -dissatisfied -gan -telecommunication -mingled -sobbed -6000 -controversies -outdated -taxis -##raus -fright -slams -##lham -##fect -##tten -detectors -fetal -tanned -##uw -fray -goth -olympian -skipping -mandates -scratches -sheng -unspoken -hyundai -tracey -hotspur -restrictive -##buch -americana -mundo -##bari -burroughs -diva -vulcan -##6th -distinctions -thumping -##ngen -mikey -sheds -fide -rescues -springsteen -vested -valuation -##ece -##ely -pinnacle -rake -sylvie -##edo -almond -quivering -##irus -alteration -faltered -##wad -51st -hydra -ticked -##kato -recommends -##dicated -antigua -arjun -stagecoach -wilfred -trickle -pronouns -##pon -aryan -nighttime -##anian -gall -pea -stitch -##hei -leung -milos -##dini -eritrea -nexus -starved -snowfall -kant -parasitic -cot -discus -hana -strikers -appleton -kitchens -##erina -##partisan -##itha -##vius -disclose -metis -##channel -1701 -tesla -##vera -fitch -1735 -blooded -##tila -decimal -##tang -##bai -cyclones -eun -bottled -peas -pensacola -basha -bolivian -crabs -boil -lanterns -partridge -roofed -1645 -necks -##phila -opined -patting -##kla -##lland -chuckles -volta -whereupon -##nche -devout -euroleague -suicidal -##dee -inherently -involuntary -knitting -nasser -##hide -puppets -colourful -courageous -southend -stills -miraculous -hodgson -richer -rochdale -ethernet -greta -uniting -prism -umm -##haya -##itical -##utation -deterioration -pointe -prowess -##ropriation -lids -scranton -billings -subcontinent -##koff -##scope -brute -kellogg -psalms -degraded -##vez -stanisław -##ructured -ferreira -pun -astonishing -gunnar -##yat -arya -prc -gottfried -##tight -excursion -##ographer -dina -##quil -##nare -huffington -illustrious -wilbur -gundam -verandah -##zard -naacp -##odle -constructive -fjord -kade -##naud -generosity -thrilling -baseline -cayman -frankish -plastics -accommodations -zoological -##fting -cedric -qb -motorized -##dome -##otted -squealed -tackled -canucks -budgets -situ -asthma -dail -gabled -grasslands -whimpered -writhing -judgments -##65 -minnie -pv -##carbon -bananas -grille -domes -monique -odin -maguire -markham -tierney -##estra -##chua -libel -poke -speedy -atrium -laval -notwithstanding -##edly -fai -kala -##sur -robb -##sma -listings -luz -supplementary -tianjin -##acing -enzo -jd -ric -scanner -croats -transcribed -##49 -arden -cv -##hair -##raphy -##lver -##uy -357 -seventies -staggering -alam -horticultural -hs -regression -timbers -blasting -##ounded -montagu -manipulating -##cit -catalytic -1550 -troopers -##meo -condemnation -fitzpatrick -##oire -##roved -inexperienced -1670 -castes -##lative -outing -314 -dubois -flicking -quarrel -ste -learners -1625 -iq -whistled -##class -282 -classify -tariffs -temperament -355 -folly -liszt -##yles -immersed -jordanian -ceasefire -apparel -extras -maru -fished -##bio -harta -stockport -assortment -craftsman -paralysis -transmitters -##cola -blindness -##wk -fatally -proficiency -solemnly -##orno -repairing -amore -groceries -ultraviolet -##chase -schoolhouse -##tua -resurgence -nailed -##otype -##× -ruse -saliva -diagrams -##tructing -albans -rann -thirties -1b -antennas -hilarious -cougars -paddington -stats -##eger -breakaway -ipod -reza -authorship -prohibiting -scoffed -##etz -##ttle -conscription -defected -trondheim -##fires -ivanov -keenan -##adan -##ciful -##fb -##slow -locating -##ials -##tford -cadiz -basalt -blankly -interned -rags -rattling -##tick -carpathian -reassured -sync -bum -guildford -iss -staunch -##onga -astronomers -sera -sofie -emergencies -susquehanna -##heard -duc -mastery -vh1 -williamsburg -bayer -buckled -craving -##khan -##rdes -bloomington -##write -alton -barbecue -##bians -justine -##hri -##ndt -delightful -smartphone -newtown -photon -retrieval -peugeot -hissing -##monium -##orough -flavors -lighted -relaunched -tainted -##games -##lysis -anarchy -microscopic -hopping -adept -evade -evie -##beau -inhibit -sinn -adjustable -hurst -intuition -wilton -cisco -44th -lawful -lowlands -stockings -thierry -##dalen -##hila -##nai -fates -prank -tb -maison -lobbied -provocative -1724 -4a -utopia -##qual -carbonate -gujarati -purcell -##rford -curtiss -##mei -overgrown -arenas -mediation -swallows -##rnik -respectful -turnbull -##hedron -##hope -alyssa -ozone -##ʻi -ami -gestapo -johansson -snooker -canteen -cuff -declines -empathy -stigma -##ags -##iner -##raine -taxpayers -gui -volga -##wright -##copic -lifespan -overcame -tattooed -enactment -giggles -##ador -##camp -barrington -bribe -obligatory -orbiting -peng -##enas -elusive -sucker -##vating -cong -hardship -empowered -anticipating -estrada -cryptic -greasy -detainees -planck -sudbury -plaid -dod -marriott -kayla -##ears -##vb -##zd -mortally -##hein -cognition -radha -319 -liechtenstein -meade -richly -argyle -harpsichord -liberalism -trumpets -lauded -tyrant -salsa -tiled -lear -promoters -reused -slicing -trident -##chuk -##gami -##lka -cantor -checkpoint -##points -gaul -leger -mammalian -##tov -##aar -##schaft -doha -frenchman -nirvana -##vino -delgado -headlining -##eron -##iography -jug -tko -1649 -naga -intersections -##jia -benfica -nawab -##suka -ashford -gulp -##deck -##vill -##rug -brentford -frazier -pleasures -dunne -potsdam -shenzhen -dentistry -##tec -flanagan -##dorff -##hear -chorale -dinah -prem -quezon -##rogated -relinquished -sutra -terri -##pani -flaps -##rissa -poly -##rnet -homme -aback -##eki -linger -womb -##kson -##lewood -doorstep -orthodoxy -threaded -westfield -##rval -dioceses -fridays -subsided -##gata -loyalists -##biotic -##ettes -letterman -lunatic -prelate -tenderly -invariably -souza -thug -winslow -##otide -furlongs -gogh -jeopardy -##runa -pegasus -##umble -humiliated -standalone -tagged -##roller -freshmen -klan -##bright -attaining -initiating -transatlantic -logged -viz -##uance -1723 -combatants -intervening -stephane -chieftain -despised -grazed -317 -cdc -galveston -godzilla -macro -simulate -##planes -parades -##esses -960 -##ductive -##unes -equator -overdose -##cans -##hosh -##lifting -joshi -epstein -sonora -treacherous -aquatics -manchu -responsive -##sation -supervisory -##christ -##llins -##ibar -##balance -##uso -kimball -karlsruhe -mab -##emy -ignores -phonetic -reuters -spaghetti -820 -almighty -danzig -rumbling -tombstone -designations -lured -outset -##felt -supermarkets -##wt -grupo -kei -kraft -susanna -##blood -comprehension -genealogy -##aghan -##verted -redding -##ythe -1722 -bowing -##pore -##roi -lest -sharpened -fulbright -valkyrie -sikhs -##unds -swans -bouquet -merritt -##tage -##venting -commuted -redhead -clerks -leasing -cesare -dea -hazy -##vances -fledged -greenfield -servicemen -##gical -armando -blackout -dt -sagged -downloadable -intra -potion -pods -##4th -##mism -xp -attendants -gambia -stale -##ntine -plump -asteroids -rediscovered -buds -flea -hive -##neas -1737 -classifications -debuts -##eles -olympus -scala -##eurs -##gno -##mute -hummed -sigismund -visuals -wiggled -await -pilasters -clench -sulfate -##ances -bellevue -enigma -trainee -snort -##sw -clouded -denim -##rank -##rder -churning -hartman -lodges -riches -sima -##missible -accountable -socrates -regulates -mueller -##cr -1702 -avoids -solids -himalayas -nutrient -pup -##jevic -squat -fades -nec -##lates -##pina -##rona -##ου -privateer -tequila -##gative -##mpton -apt -hornet -immortals -##dou -asturias -cleansing -dario -##rries -##anta -etymology -servicing -zhejiang -##venor -##nx -horned -erasmus -rayon -relocating -£10 -##bags -escalated -promenade -stubble -2010s -artisans -axial -liquids -mora -sho -yoo -##tsky -bundles -oldies -##nally -notification -bastion -##ths -sparkle -##lved -1728 -leash -pathogen -highs -##hmi -immature -880 -gonzaga -ignatius -mansions -monterrey -sweets -bryson -##loe -polled -regatta -brightest -pei -rosy -squid -hatfield -payroll -addict -meath -cornerback -heaviest -lodging -##mage -capcom -rippled -##sily -barnet -mayhem -ymca -snuggled -rousseau -##cute -blanchard -284 -fragmented -leighton -chromosomes -risking -##md -##strel -##utter -corinne -coyotes -cynical -hiroshi -yeomanry -##ractive -ebook -grading -mandela -plume -agustin -magdalene -##rkin -bea -femme -trafford -##coll -##lun -##tance -52nd -fourier -upton -##mental -camilla -gust -iihf -islamabad -longevity -##kala -feldman -netting -##rization -endeavour -foraging -mfa -orr -##open -greyish -contradiction -graz -##ruff -handicapped -marlene -tweed -oaxaca -spp -campos -miocene -pri -configured -cooks -pluto -cozy -pornographic -##entes -70th -fairness -glided -jonny -lynne -rounding -sired -##emon -##nist -remade -uncover -##mack -complied -lei -newsweek -##jured -##parts -##enting -##pg -293 -finer -guerrillas -athenian -deng -disused -stepmother -accuse -gingerly -seduction -521 -confronting -##walker -##going -gora -nostalgia -sabres -virginity -wrenched -##minated -syndication -wielding -eyre -##56 -##gnon -##igny -behaved -taxpayer -sweeps -##growth -childless -gallant -##ywood -amplified -geraldine -scrape -##ffi -babylonian -fresco -##rdan -##kney -##position -1718 -restricting -tack -fukuoka -osborn -selector -partnering -##dlow -318 -gnu -kia -tak -whitley -gables -##54 -##mania -mri -softness -immersion -##bots -##evsky -1713 -chilling -insignificant -pcs -##uis -elites -lina -purported -supplemental -teaming -##americana -##dding -##inton -proficient -rouen -##nage -##rret -niccolo -selects -##bread -fluffy -1621 -gruff -knotted -mukherjee -polgara -thrash -nicholls -secluded -smoothing -thru -corsica -loaf -whitaker -inquiries -##rrier -##kam -indochina -289 -marlins -myles -peking -##tea -extracts -pastry -superhuman -connacht -vogel -##ditional -##het -##udged -##lash -gloss -quarries -refit -teaser -##alic -##gaon -20s -materialized -sling -camped -pickering -tung -tracker -pursuant -##cide -cranes -soc -##cini -##typical -##viere -anhalt -overboard -workout -chores -fares -orphaned -stains -##logie -fenton -surpassing -joyah -triggers -##itte -grandmaster -##lass -##lists -clapping -fraudulent -ledger -nagasaki -##cor -##nosis -##tsa -eucalyptus -tun -##icio -##rney -##tara -dax -heroism -ina -wrexham -onboard -unsigned -##dates -moshe -galley -winnie -droplets -exiles -praises -watered -noodles -##aia -fein -adi -leland -multicultural -stink -bingo -comets -erskine -modernized -canned -constraint -domestically -chemotherapy -featherweight -stifled -##mum -darkly -irresistible -refreshing -hasty -isolate -##oys -kitchener -planners -##wehr -cages -yarn -implant -toulon -elects -childbirth -yue -##lind -##lone -cn -rightful -sportsman -junctions -remodeled -specifies -##rgh -291 -##oons -complimented -##urgent -lister -ot -##logic -bequeathed -cheekbones -fontana -gabby -##dial -amadeus -corrugated -maverick -resented -triangles -##hered -##usly -nazareth -tyrol -1675 -assent -poorer -sectional -aegean -##cous -296 -nylon -ghanaian -##egorical -##weig -cushions -forbid -fusiliers -obstruction -somerville -##scia -dime -earrings -elliptical -leyte -oder -polymers -timmy -atm -midtown -piloted -settles -continual -externally -mayfield -##uh -enrichment -henson -keane -persians -1733 -benji -braden -pep -324 -##efe -contenders -pepsi -valet -##isches -298 -##asse -##earing -goofy -stroll -##amen -authoritarian -occurrences -adversary -ahmedabad -tangent -toppled -dorchester -1672 -modernism -marxism -islamist -charlemagne -exponential -racks -unicode -brunette -mbc -pic -skirmish -##bund -##lad -##powered -##yst -hoisted -messina -shatter -##ctum -jedi -vantage -##music -##neil -clemens -mahmoud -corrupted -authentication -lowry -nils -##washed -omnibus -wounding -jillian -##itors -##opped -serialized -narcotics -handheld -##arm -##plicity -intersecting -stimulating -##onis -crate -fellowships -hemingway -casinos -climatic -fordham -copeland -drip -beatty -leaflets -robber -brothel -madeira -##hedral -sphinx -ultrasound -##vana -valor -forbade -leonid -villas -##aldo -duane -marquez -##cytes -disadvantaged -forearms -kawasaki -reacts -consular -lax -uncles -uphold -##hopper -concepcion -dorsey -lass -##izan -arching -passageway -1708 -researches -tia -internationals -##graphs -##opers -distinguishes -javanese -divert -##uven -plotted -##listic -##rwin -##erik -##tify -affirmative -signifies -validation -##bson -kari -felicity -georgina -zulu -##eros -##rained -##rath -overcoming -##dot -argyll -##rbin -1734 -chiba -ratification -windy -earls -parapet -##marks -hunan -pristine -astrid -punta -##gart -brodie -##kota -##oder -malaga -minerva -rouse -##phonic -bellowed -pagoda -portals -reclamation -##gur -##odies -##⁄₄ -parentheses -quoting -allergic -palette -showcases -benefactor -heartland -nonlinear -##tness -bladed -cheerfully -scans -##ety -##hone -1666 -girlfriends -pedersen -hiram -sous -##liche -##nator -1683 -##nery -##orio -##umen -bobo -primaries -smiley -##cb -unearthed -uniformly -fis -metadata -1635 -ind -##oted -recoil -##titles -##tura -##ια -406 -hilbert -jamestown -mcmillan -tulane -seychelles -##frid -antics -coli -fated -stucco -##grants -1654 -bulky -accolades -arrays -caledonian -carnage -optimism -puebla -##tative -##cave -enforcing -rotherham -seo -dunlop -aeronautics -chimed -incline -zoning -archduke -hellenistic -##oses -##sions -candi -thong -##ople -magnate -rustic -##rsk -projective -slant -##offs -danes -hollis -vocalists -##ammed -congenital -contend -gesellschaft -##ocating -##pressive -douglass -quieter -##cm -##kshi -howled -salim -spontaneously -townsville -buena -southport -##bold -kato -1638 -faerie -stiffly -##vus -##rled -297 -flawless -realising -taboo -##7th -bytes -straightening -356 -jena -##hid -##rmin -cartwright -berber -bertram -soloists -411 -noses -417 -coping -fission -hardin -inca -##cen -1717 -mobilized -vhf -##raf -biscuits -curate -##85 -##anial -331 -gaunt -neighbourhoods -1540 -##abas -blanca -bypassed -sockets -behold -coincidentally -##bane -nara -shave -splinter -terrific -##arion -##erian -commonplace -juris -redwood -waistband -boxed -caitlin -fingerprints -jennie -naturalized -##ired -balfour -craters -jody -bungalow -hugely -quilt -glitter -pigeons -undertaker -bulging -constrained -goo -##sil -##akh -assimilation -reworked -##person -persuasion -##pants -felicia -##cliff -##ulent -1732 -explodes -##dun -##inium -##zic -lyman -vulture -hog -overlook -begs -northwards -ow -spoil -##urer -fatima -favorably -accumulate -sargent -sorority -corresponded -dispersal -kochi -toned -##imi -##lita -internacional -newfound -##agger -##lynn -##rigue -booths -peanuts -##eborg -medicare -muriel -nur -##uram -crates -millennia -pajamas -worsened -##breakers -jimi -vanuatu -yawned -##udeau -carousel -##hony -hurdle -##ccus -##mounted -##pod -rv -##eche -airship -ambiguity -compulsion -recapture -##claiming -arthritis -##osomal -1667 -asserting -ngc -sniffing -dade -discontent -glendale -ported -##amina -defamation -rammed -##scent -fling -livingstone -##fleet -875 -##ppy -apocalyptic -comrade -lcd -##lowe -cessna -eine -persecuted -subsistence -demi -hoop -reliefs -710 -coptic -progressing -stemmed -perpetrators -1665 -priestess -##nio -dobson -ebony -rooster -itf -tortricidae -##bbon -##jian -cleanup -##jean -##øy -1721 -eighties -taxonomic -holiness -##hearted -##spar -antilles -showcasing -stabilized -##nb -gia -mascara -michelangelo -dawned -##uria -##vinsky -extinguished -fitz -grotesque -£100 -##fera -##loid -##mous -barges -neue -throbbed -cipher -johnnie -##a1 -##mpt -outburst -##swick -spearheaded -administrations -c1 -heartbreak -pixels -pleasantly -##enay -lombardy -plush -##nsed -bobbie -##hly -reapers -tremor -xiang -minogue -substantive -hitch -barak -##wyl -kwan -##encia -910 -obscene -elegance -indus -surfer -bribery -conserve -##hyllum -##masters -horatio -##fat -apes -rebound -psychotic -##pour -iteration -##mium -##vani -botanic -horribly -antiques -dispose -paxton -##hli -##wg -timeless -1704 -disregard -engraver -hounds -##bau -##version -looted -uno -facilitates -groans -masjid -rutland -antibody -disqualification -decatur -footballers -quake -slacks -48th -rein -scribe -stabilize -commits -exemplary -tho -##hort -##chison -pantry -traversed -##hiti -disrepair -identifiable -vibrated -baccalaureate -##nnis -csa -interviewing -##iensis -##raße -greaves -wealthiest -343 -classed -jogged -£5 -##58 -##atal -illuminating -knicks -respecting -##uno -scrubbed -##iji -##dles -kruger -moods -growls -raider -silvia -chefs -kam -vr -cree -percival -##terol -gunter -counterattack -defiant -henan -ze -##rasia -##riety -equivalence -submissions -##fra -##thor -bautista -mechanically -##heater -cornice -herbal -templar -##mering -outputs -ruining -ligand -renumbered -extravagant -mika -blockbuster -eta -insurrection -##ilia -darkening -ferocious -pianos -strife -kinship -##aer -melee -##anor -##iste -##may -##oue -decidedly -weep -##jad -##missive -##ppel -354 -puget -unease -##gnant -1629 -hammering -kassel -ob -wessex -##lga -bromwich -egan -paranoia -utilization -##atable -##idad -contradictory -provoke -##ols -##ouring -##tangled -knesset -##very -##lette -plumbing -##sden -##¹ -greensboro -occult -sniff -338 -zev -beaming -gamer -haggard -mahal -##olt -##pins -mendes -utmost -briefing -gunnery -##gut -##pher -##zh -##rok -1679 -khalifa -sonya -##boot -principals -urbana -wiring -##liffe -##minating -##rrado -dahl -nyu -skepticism -np -townspeople -ithaca -lobster -somethin -##fur -##arina -##−1 -freighter -zimmerman -biceps -contractual -##herton -amend -hurrying -subconscious -##anal -336 -meng -clermont -spawning -##eia -##lub -dignitaries -impetus -snacks -spotting -twigs -##bilis -##cz -##ouk -libertadores -nic -skylar -##aina -##firm -gustave -asean -##anum -dieter -legislatures -flirt -bromley -trolls -umar -##bbies -##tyle -blah -parc -bridgeport -crank -negligence -##nction -46th -constantin -molded -bandages -seriousness -00pm -siegel -carpets -compartments -upbeat -statehood -##dner -##edging -marko -730 -platt -##hane -paving -##iy -1738 -abbess -impatience -limousine -nbl -##talk -441 -lucille -mojo -nightfall -robbers -##nais -karel -brisk -calves -replicate -ascribed -telescopes -##olf -intimidated -##reen -ballast -specialization -##sit -aerodynamic -caliphate -rainer -visionary -##arded -epsilon -##aday -##onte -aggregation -auditory -boosted -reunification -kathmandu -loco -robyn -402 -acknowledges -appointing -humanoid -newell -redeveloped -restraints -##tained -barbarians -chopper -1609 -italiana -##lez -##lho -investigates -wrestlemania -##anies -##bib -690 -##falls -creaked -dragoons -gravely -minions -stupidity -volley -##harat -##week -musik -##eries -##uously -fungal -massimo -semantics -malvern -##ahl -##pee -discourage -embryo -imperialism -1910s -profoundly -##ddled -jiangsu -sparkled -stat -##holz -sweatshirt -tobin -##iction -sneered -##cheon -##oit -brit -causal -smyth -##neuve -diffuse -perrin -silvio -##ipes -##recht -detonated -iqbal -selma -##nism -##zumi -roasted -##riders -tay -##ados -##mament -##mut -##rud -840 -completes -nipples -cfa -flavour -hirsch -##laus -calderon -sneakers -moravian -##ksha -1622 -rq -294 -##imeters -bodo -##isance -##pre -##ronia -anatomical -excerpt -##lke -dh -kunst -##tablished -##scoe -biomass -panted -unharmed -gael -housemates -montpellier -##59 -coa -rodents -tonic -hickory -singleton -##taro -451 -1719 -aldo -breaststroke -dempsey -och -rocco -##cuit -merton -dissemination -midsummer -serials -##idi -haji -polynomials -##rdon -gs -enoch -prematurely -shutter -taunton -£3 -##grating -##inates -archangel -harassed -##asco -326 -archway -dazzling -##ecin -1736 -sumo -wat -##kovich -1086 -honneur -##ently -##nostic -##ttal -##idon -1605 -403 -1716 -blogger -rents -##gnan -hires -##ikh -##dant -howie -##rons -handler -retracted -shocks -1632 -arun -duluth -kepler -trumpeter -##lary -peeking -seasoned -trooper -##mara -laszlo -##iciencies -##rti -heterosexual -##inatory -##ssion -indira -jogging -##inga -##lism -beit -dissatisfaction -malice -##ately -nedra -peeling -##rgeon -47th -stadiums -475 -vertigo -##ains -iced -restroom -##plify -##tub -illustrating -pear -##chner -##sibility -inorganic -rappers -receipts -watery -##kura -lucinda -##oulos -reintroduced -##8th -##tched -gracefully -saxons -nutritional -wastewater -rained -favourites -bedrock -fisted -hallways -likeness -upscale -##lateral -1580 -blinds -prequel -##pps -##tama -deter -humiliating -restraining -tn -vents -1659 -laundering -recess -rosary -tractors -coulter -federer -##ifiers -##plin -persistence -##quitable -geschichte -pendulum -quakers -##beam -bassett -pictorial -buffet -koln -##sitor -drills -reciprocal -shooters -##57 -##cton -##tees -converge -pip -dmitri -donnelly -yamamoto -aqua -azores -demographics -hypnotic -spitfire -suspend -wryly -roderick -##rran -sebastien -##asurable -mavericks -##fles -##200 -himalayan -prodigy -##iance -transvaal -demonstrators -handcuffs -dodged -mcnamara -sublime -1726 -crazed -##efined -##till -ivo -pondered -reconciled -shrill -sava -##duk -bal -cad -heresy -jaipur -goran -##nished -341 -lux -shelly -whitehall -##hre -israelis -peacekeeping -##wled -1703 -demetrius -ousted -##arians -##zos -beale -anwar -backstroke -raged -shrinking -cremated -##yck -benign -towing -wadi -darmstadt -landfill -parana -soothe -colleen -sidewalks -mayfair -tumble -hepatitis -ferrer -superstructure -##gingly -##urse -##wee -anthropological -translators -##mies -closeness -hooves -##pw -mondays -##roll -##vita -landscaping -##urized -purification -sock -thorns -thwarted -jalan -tiberius -##taka -saline -##rito -confidently -khyber -sculptors -##ij -brahms -hammersmith -inspectors -battista -fivb -fragmentation -hackney -##uls -arresting -exercising -antoinette -bedfordshire -##zily -dyed -##hema -1656 -racetrack -variability -##tique -1655 -austrians -deteriorating -madman -theorists -aix -lehman -weathered -1731 -decreed -eruptions -1729 -flaw -quinlan -sorbonne -flutes -nunez -1711 -adored -downwards -fable -rasped -1712 -moritz -mouthful -renegade -shivers -stunts -dysfunction -restrain -translit -327 -pancakes -##avio -##cision -##tray -351 -vial -##lden -bain -##maid -##oxide -chihuahua -malacca -vimes -##rba -##rnier -1664 -donnie -plaques -##ually -337 -bangs -floppy -huntsville -loretta -nikolay -##otte -eater -handgun -ubiquitous -##hett -eras -zodiac -1634 -##omorphic -1820s -##zog -cochran -##bula -##lithic -warring -##rada -dalai -excused -blazers -mcconnell -reeling -bot -este -##abi -geese -hoax -taxon -##bla -guitarists -##icon -condemning -hunts -inversion -moffat -taekwondo -##lvis -1624 -stammered -##rest -##rzy -sousa -fundraiser -marylebone -navigable -uptown -cabbage -daniela -salman -shitty -whimper -##kian -##utive -programmers -protections -rm -##rmi -##rued -forceful -##enes -fuss -##tao -##wash -brat -oppressive -reykjavik -spartak -ticking -##inkles -##kiewicz -adolph -horst -maui -protege -straighten -cpc -landau -concourse -clements -resultant -##ando -imaginative -joo -reactivated -##rem -##ffled -##uising -consultative -##guide -flop -kaitlyn -mergers -parenting -somber -##vron -supervise -vidhan -##imum -courtship -exemplified -harmonies -medallist -refining -##rrow -##ка -amara -##hum -780 -goalscorer -sited -overshadowed -rohan -displeasure -secretive -multiplied -osman -##orth -engravings -padre -##kali -##veda -miniatures -mis -##yala -clap -pali -rook -##cana -1692 -57th -antennae -astro -oskar -1628 -bulldog -crotch -hackett -yucatan -##sure -amplifiers -brno -ferrara -migrating -##gree -thanking -turing -##eza -mccann -ting -andersson -onslaught -gaines -ganga -incense -standardization -##mation -sentai -scuba -stuffing -turquoise -waivers -alloys -##vitt -regaining -vaults -##clops -##gizing -digger -furry -memorabilia -probing -##iad -payton -rec -deutschland -filippo -opaque -seamen -zenith -afrikaans -##filtration -disciplined -inspirational -##merie -banco -confuse -grafton -tod -##dgets -championed -simi -anomaly -biplane -##ceptive -electrode -##para -1697 -cleavage -crossbow -swirl -informant -##lars -##osta -afi -bonfire -spec -##oux -lakeside -slump -##culus -##lais -##qvist -##rrigan -1016 -facades -borg -inwardly -cervical -xl -pointedly -050 -stabilization -##odon -chests -1699 -hacked -ctv -orthogonal -suzy -##lastic -gaulle -jacobite -rearview -##cam -##erted -ashby -##drik -##igate -##mise -##zbek -affectionately -canine -disperse -latham -##istles -##ivar -spielberg -##orin -##idium -ezekiel -cid -##sg -durga -middletown -##cina -customized -frontiers -harden -##etano -##zzy -1604 -bolsheviks -##66 -coloration -yoko -##bedo -briefs -slabs -debra -liquidation -plumage -##oin -blossoms -dementia -subsidy -1611 -proctor -relational -jerseys -parochial -ter -##ici -esa -peshawar -cavalier -loren -cpi -idiots -shamrock -1646 -dutton -malabar -mustache -##endez -##ocytes -referencing -terminates -marche -yarmouth -##sop -acton -mated -seton -subtly -baptised -beige -extremes -jolted -kristina -telecast -##actic -safeguard -waldo -##baldi -##bular -endeavors -sloppy -subterranean -##ensburg -##itung -delicately -pigment -tq -##scu -1626 -##ound -collisions -coveted -herds -##personal -##meister -##nberger -chopra -##ricting -abnormalities -defective -galician -lucie -##dilly -alligator -likened -##genase -burundi -clears -complexion -derelict -deafening -diablo -fingered -champaign -dogg -enlist -isotope -labeling -mrna -##erre -brilliance -marvelous -##ayo -1652 -crawley -ether -footed -dwellers -deserts -hamish -rubs -warlock -skimmed -##lizer -870 -buick -embark -heraldic -irregularities -##ajan -kiara -##kulam -##ieg -antigen -kowalski -##lge -oakley -visitation -##mbit -vt -##suit -1570 -murderers -##miento -##rites -chimneys -##sling -condemn -custer -exchequer -havre -##ghi -fluctuations -##rations -dfb -hendricks -vaccines -##tarian -nietzsche -biking -juicy -##duced -brooding -scrolling -selangor -##ragan -352 -annum -boomed -seminole -sugarcane -##dna -departmental -dismissing -innsbruck -arteries -ashok -batavia -daze -kun -overtook -##rga -##tlan -beheaded -gaddafi -holm -electronically -faulty -galilee -fractures -kobayashi -##lized -gunmen -magma -aramaic -mala -eastenders -inference -messengers -bf -##qu -407 -bathrooms -##vere -1658 -flashbacks -ideally -misunderstood -##jali -##weather -mendez -##grounds -505 -uncanny -##iii -1709 -friendships -##nbc -sacrament -accommodated -reiterated -logistical -pebbles -thumped -##escence -administering -decrees -drafts -##flight -##cased -##tula -futuristic -picket -intimidation -winthrop -##fahan -interfered -339 -afar -francoise -morally -uta -cochin -croft -dwarfs -##bruck -##dents -##nami -biker -##hner -##meral -nano -##isen -##ometric -##pres -##ан -brightened -meek -parcels -securely -gunners -##jhl -##zko -agile -hysteria -##lten -##rcus -bukit -champs -chevy -cuckoo -leith -sadler -theologians -welded -##section -1663 -jj -plurality -xander -##rooms -##formed -shredded -temps -intimately -pau -tormented -##lok -##stellar -1618 -charred -ems -essen -##mmel -alarms -spraying -ascot -blooms -twinkle -##abia -##apes -internment -obsidian -##chaft -snoop -##dav -##ooping -malibu -##tension -quiver -##itia -hays -mcintosh -travers -walsall -##ffie -1623 -beverley -schwarz -plunging -structurally -m3 -rosenthal -vikram -##tsk -770 -ghz -##onda -##tiv -chalmers -groningen -pew -reckon -unicef -##rvis -55th -##gni -1651 -sulawesi -avila -cai -metaphysical -screwing -turbulence -##mberg -augusto -samba -56th -baffled -momentary -toxin -##urian -##wani -aachen -condoms -dali -steppe -##3d -##app -##oed -##year -adolescence -dauphin -electrically -inaccessible -microscopy -nikita -##ega -atv -##cel -##enter -##oles -##oteric -##ы -accountants -punishments -wrongly -bribes -adventurous -clinch -flinders -southland -##hem -##kata -gough -##ciency -lads -soared -##ה -undergoes -deformation -outlawed -rubbish -##arus -##mussen -##nidae -##rzburg -arcs -##ingdon -##tituted -1695 -wheelbase -wheeling -bombardier -campground -zebra -##lices -##oj -##bain -lullaby -##ecure -donetsk -wylie -grenada -##arding -##ης -squinting -eireann -opposes -##andra -maximal -runes -##broken -##cuting -##iface -##ror -##rosis -additive -britney -adultery -triggering -##drome -detrimental -aarhus -containment -jc -swapped -vichy -##ioms -madly -##oric -##rag -brant -##ckey -##trix -1560 -1612 -broughton -rustling -##stems -##uder -asbestos -mentoring -##nivorous -finley -leaps -##isan -apical -pry -slits -substitutes -##dict -intuitive -fantasia -insistent -unreasonable -##igen -##vna -domed -hannover -margot -ponder -##zziness -impromptu -jian -lc -rampage -stemming -##eft -andrey -gerais -whichever -amnesia -appropriated -anzac -clicks -modifying -ultimatum -cambrian -maids -verve -yellowstone -##mbs -conservatoire -##scribe -adherence -dinners -spectra -imperfect -mysteriously -sidekick -tatar -tuba -##aks -##ifolia -distrust -##athan -##zle -c2 -ronin -zac -##pse -celaena -instrumentalist -scents -skopje -##mbling -comical -compensated -vidal -condor -intersect -jingle -wavelengths -##urrent -mcqueen -##izzly -carp -weasel -422 -kanye -militias -postdoctoral -eugen -gunslinger -##ɛ -faux -hospice -##for -appalled -derivation -dwarves -##elis -dilapidated -##folk -astoria -philology -##lwyn -##otho -##saka -inducing -philanthropy -##bf -##itative -geek -markedly -sql -##yce -bessie -indices -rn -##flict -495 -frowns -resolving -weightlifting -tugs -cleric -contentious -1653 -mania -rms -##miya -##reate -##ruck -##tucket -bien -eels -marek -##ayton -##cence -discreet -unofficially -##ife -leaks -##bber -1705 -332 -dung -compressor -hillsborough -pandit -shillings -distal -##skin -381 -##tat -##you -nosed -##nir -mangrove -undeveloped -##idia -textures -##inho -##500 -##rise -ae -irritating -nay -amazingly -bancroft -apologetic -compassionate -kata -symphonies -##lovic -airspace -##lch -930 -gifford -precautions -fulfillment -sevilla -vulgar -martinique -##urities -looting -piccolo -tidy -##dermott -quadrant -armchair -incomes -mathematicians -stampede -nilsson -##inking -##scan -foo -quarterfinal -##ostal -shang -shouldered -squirrels -##owe -344 -vinegar -##bner -##rchy -##systems -delaying -##trics -ars -dwyer -rhapsody -sponsoring -##gration -bipolar -cinder -starters -##olio -##urst -421 -signage -##nty -aground -figurative -mons -acquaintances -duets -erroneously -soyuz -elliptic -recreated -##cultural -##quette -##ssed -##tma -##zcz -moderator -scares -##itaire -##stones -##udence -juniper -sighting -##just -##nsen -britten -calabria -ry -bop -cramer -forsyth -stillness -##л -airmen -gathers -unfit -##umber -##upt -taunting -##rip -seeker -streamlined -##bution -holster -schumann -tread -vox -##gano -##onzo -strive -dil -reforming -covent -newbury -predicting -##orro -decorate -tre -##puted -andover -ie -asahi -dept -dunkirk -gills -##tori -buren -huskies -##stis -##stov -abstracts -bets -loosen -##opa -1682 -yearning -##glio -##sir -berman -effortlessly -enamel -napoli -persist -##peration -##uez -attache -elisa -b1 -invitations -##kic -accelerating -reindeer -boardwalk -clutches -nelly -polka -starbucks -##kei -adamant -huey -lough -unbroken -adventurer -embroidery -inspecting -stanza -##ducted -naia -taluka -##pone -##roids -chases -deprivation -florian -##jing -##ppet -earthly -##lib -##ssee -colossal -foreigner -vet -freaks -patrice -rosewood -triassic -upstate -##pkins -dominates -ata -chants -ks -vo -##400 -##bley -##raya -##rmed -555 -agra -infiltrate -##ailing -##ilation -##tzer -##uppe -##werk -binoculars -enthusiast -fujian -squeak -##avs -abolitionist -almeida -boredom -hampstead -marsden -rations -##ands -inflated -334 -bonuses -rosalie -patna -##rco -329 -detachments -penitentiary -54th -flourishing -woolf -##dion -##etched -papyrus -##lster -##nsor -##toy -bobbed -dismounted -endelle -inhuman -motorola -tbs -wince -wreath -##ticus -hideout -inspections -sanjay -disgrace -infused -pudding -stalks -##urbed -arsenic -leases -##hyl -##rrard -collarbone -##waite -##wil -dowry -##bant -##edance -genealogical -nitrate -salamanca -scandals -thyroid -necessitated -##! -##" -### -##$ -##% -##& -##' -##( -##) -##* -##+ -##, -##- -##. -##/ -##: -##; -##< -##= -##> -##? -##@ -##[ -##\ -##] -##^ -##_ -##` -##{ -##| -##} -##~ -##¡ -##¢ -##£ -##¤ -##¥ -##¦ -##§ -##¨ -##© -##ª -##« -##¬ -##® -##± -##´ -##µ -##¶ -##· -##º -##» -##¼ -##¾ -##¿ -##æ -##ð -##÷ -##þ -##đ -##ħ -##ŋ -##œ -##ƒ -##ɐ -##ɑ -##ɒ -##ɔ -##ɕ -##ə -##ɡ -##ɣ -##ɨ -##ɪ -##ɫ -##ɬ -##ɯ -##ɲ -##ɴ -##ɹ -##ɾ -##ʀ -##ʁ -##ʂ -##ʃ -##ʉ -##ʊ -##ʋ -##ʌ -##ʎ -##ʐ -##ʑ -##ʒ -##ʔ -##ʰ -##ʲ -##ʳ -##ʷ -##ʸ -##ʻ -##ʼ -##ʾ -##ʿ -##ˈ -##ˡ -##ˢ -##ˣ -##ˤ -##β -##γ -##δ -##ε -##ζ -##θ -##κ -##λ -##μ -##ξ -##ο -##π -##ρ -##σ -##τ -##υ -##φ -##χ -##ψ -##ω -##б -##г -##д -##ж -##з -##м -##п -##с -##у -##ф -##х -##ц -##ч -##ш -##щ -##ъ -##э -##ю -##ђ -##є -##і -##ј -##љ -##њ -##ћ -##ӏ -##ա -##բ -##գ -##դ -##ե -##թ -##ի -##լ -##կ -##հ -##մ -##յ -##ն -##ո -##պ -##ս -##վ -##տ -##ր -##ւ -##ք -##־ -##א -##ב -##ג -##ד -##ו -##ז -##ח -##ט -##י -##ך -##כ -##ל -##ם -##מ -##ן -##נ -##ס -##ע -##ף -##פ -##ץ -##צ -##ק -##ר -##ש -##ת -##، -##ء -##ب -##ت -##ث -##ج -##ح -##خ -##ذ -##ز -##س -##ش -##ص -##ض -##ط -##ظ -##ع -##غ -##ـ -##ف -##ق -##ك -##و -##ى -##ٹ -##پ -##چ -##ک -##گ -##ں -##ھ -##ہ -##ے -##अ -##आ -##उ -##ए -##क -##ख -##ग -##च -##ज -##ट -##ड -##ण -##त -##थ -##द -##ध -##न -##प -##ब -##भ -##म -##य -##र -##ल -##व -##श -##ष -##स -##ह -##ा -##ि -##ी -##ो -##। -##॥ -##ং -##অ -##আ -##ই -##উ -##এ -##ও -##ক -##খ -##গ -##চ -##ছ -##জ -##ট -##ড -##ণ -##ত -##থ -##দ -##ধ -##ন -##প -##ব -##ভ -##ম -##য -##র -##ল -##শ -##ষ -##স -##হ -##া -##ি -##ী -##ে -##க -##ச -##ட -##த -##ந -##ன -##ப -##ம -##ய -##ர -##ல -##ள -##வ -##ா -##ி -##ு -##ே -##ை -##ನ -##ರ -##ಾ -##ක -##ය -##ර -##ල -##ව -##ා -##ก -##ง -##ต -##ท -##น -##พ -##ม -##ย -##ร -##ล -##ว -##ส -##อ -##า -##เ -##་ -##། -##ག -##ང -##ད -##ན -##པ -##བ -##མ -##འ -##ར -##ལ -##ས -##မ -##ა -##ბ -##გ -##დ -##ე -##ვ -##თ -##ი -##კ -##ლ -##მ -##ნ -##ო -##რ -##ს -##ტ -##უ -##ᄀ -##ᄂ -##ᄃ -##ᄅ -##ᄆ -##ᄇ -##ᄉ -##ᄊ -##ᄋ -##ᄌ -##ᄎ -##ᄏ -##ᄐ -##ᄑ -##ᄒ -##ᅡ -##ᅢ -##ᅥ -##ᅦ -##ᅧ -##ᅩ -##ᅪ -##ᅭ -##ᅮ -##ᅯ -##ᅲ -##ᅳ -##ᅴ -##ᅵ -##ᆨ -##ᆫ -##ᆯ -##ᆷ -##ᆸ -##ᆼ -##ᴬ -##ᴮ -##ᴰ -##ᴵ -##ᴺ -##ᵀ -##ᵃ -##ᵇ -##ᵈ -##ᵉ -##ᵍ -##ᵏ -##ᵐ -##ᵒ -##ᵖ -##ᵗ -##ᵘ -##ᵣ -##ᵤ -##ᵥ -##ᶜ -##ᶠ -##‐ -##‑ -##‒ -##– -##— -##― -##‖ -##‘ -##’ -##‚ -##“ -##” -##„ -##† -##‡ -##• -##… -##‰ -##′ -##″ -##› -##‿ -##⁄ -##⁰ -##ⁱ -##⁴ -##⁵ -##⁶ -##⁷ -##⁸ -##⁹ -##⁻ -##ⁿ -##₅ -##₆ -##₇ -##₈ -##₉ -##₊ -##₍ -##₎ -##ₐ -##ₑ -##ₒ -##ₓ -##ₕ -##ₖ -##ₗ -##ₘ -##ₚ -##ₛ -##ₜ -##₤ -##₩ -##€ -##₱ -##₹ -##ℓ -##№ -##ℝ -##™ -##⅓ -##⅔ -##← -##↑ -##→ -##↓ -##↔ -##↦ -##⇄ -##⇌ -##⇒ -##∂ -##∅ -##∆ -##∇ -##∈ -##∗ -##∘ -##√ -##∞ -##∧ -##∨ -##∩ -##∪ -##≈ -##≡ -##≤ -##≥ -##⊂ -##⊆ -##⊕ -##⊗ -##⋅ -##─ -##│ -##■ -##▪ -##● -##★ -##☆ -##☉ -##♠ -##♣ -##♥ -##♦ -##♯ -##⟨ -##⟩ -##ⱼ -##⺩ -##⺼ -##⽥ -##、 -##。 -##〈 -##〉 -##《 -##》 -##「 -##」 -##『 -##』 -##〜 -##あ -##い -##う -##え -##お -##か -##き -##く -##け -##こ -##さ -##し -##す -##せ -##そ -##た -##ち -##っ -##つ -##て -##と -##な -##に -##ぬ -##ね -##の -##は -##ひ -##ふ -##へ -##ほ -##ま -##み -##む -##め -##も -##や -##ゆ -##よ -##ら -##り -##る -##れ -##ろ -##を -##ん -##ァ -##ア -##ィ -##イ -##ウ -##ェ -##エ -##オ -##カ -##キ -##ク -##ケ -##コ -##サ -##シ -##ス -##セ -##タ -##チ -##ッ -##ツ -##テ -##ト -##ナ -##ニ -##ノ -##ハ -##ヒ -##フ -##ヘ -##ホ -##マ -##ミ -##ム -##メ -##モ -##ャ -##ュ -##ョ -##ラ -##リ -##ル -##レ -##ロ -##ワ -##ン -##・ -##ー -##一 -##三 -##上 -##下 -##不 -##世 -##中 -##主 -##久 -##之 -##也 -##事 -##二 -##五 -##井 -##京 -##人 -##亻 -##仁 -##介 -##代 -##仮 -##伊 -##会 -##佐 -##侍 -##保 -##信 -##健 -##元 -##光 -##八 -##公 -##内 -##出 -##分 -##前 -##劉 -##力 -##加 -##勝 -##北 -##区 -##十 -##千 -##南 -##博 -##原 -##口 -##古 -##史 -##司 -##合 -##吉 -##同 -##名 -##和 -##囗 -##四 -##国 -##國 -##土 -##地 -##坂 -##城 -##堂 -##場 -##士 -##夏 -##外 -##大 -##天 -##太 -##夫 -##奈 -##女 -##子 -##学 -##宀 -##宇 -##安 -##宗 -##定 -##宣 -##宮 -##家 -##宿 -##寺 -##將 -##小 -##尚 -##山 -##岡 -##島 -##崎 -##川 -##州 -##巿 -##帝 -##平 -##年 -##幸 -##广 -##弘 -##張 -##彳 -##後 -##御 -##德 -##心 -##忄 -##志 -##忠 -##愛 -##成 -##我 -##戦 -##戸 -##手 -##扌 -##政 -##文 -##新 -##方 -##日 -##明 -##星 -##春 -##昭 -##智 -##曲 -##書 -##月 -##有 -##朝 -##木 -##本 -##李 -##村 -##東 -##松 -##林 -##森 -##楊 -##樹 -##橋 -##歌 -##止 -##正 -##武 -##比 -##氏 -##民 -##水 -##氵 -##氷 -##永 -##江 -##沢 -##河 -##治 -##法 -##海 -##清 -##漢 -##瀬 -##火 -##版 -##犬 -##王 -##生 -##田 -##男 -##疒 -##発 -##白 -##的 -##皇 -##目 -##相 -##省 -##真 -##石 -##示 -##社 -##神 -##福 -##禾 -##秀 -##秋 -##空 -##立 -##章 -##竹 -##糹 -##美 -##義 -##耳 -##良 -##艹 -##花 -##英 -##華 -##葉 -##藤 -##行 -##街 -##西 -##見 -##訁 -##語 -##谷 -##貝 -##貴 -##車 -##軍 -##辶 -##道 -##郎 -##郡 -##部 -##都 -##里 -##野 -##金 -##鈴 -##镇 -##長 -##門 -##間 -##阝 -##阿 -##陳 -##陽 -##雄 -##青 -##面 -##風 -##食 -##香 -##馬 -##高 -##龍 -##龸 -##fi -##fl -##！ -##（ -##） -##， -##－ -##． -##／ -##： -##？ -##～ diff --git a/tutorials/quickstart.ipynb b/tutorials/quickstart.ipynb new file mode 100644 index 00000000..00c30c93 --- /dev/null +++ b/tutorials/quickstart.ipynb @@ -0,0 +1,280 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "# 快速入门" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str}" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP.io import CSVLoader\n", + "\n", + "loader = CSVLoader(headers=('raw_sentence', 'label'), sep='\\t')\n", + "dataset = loader.load(\"./sample_data/tutorial_sample_dataset.csv\")\n", + "dataset[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str,\n", + "'sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'words': ['a', 'series', 'of', 'escapades', 'demonstrating', 'the', 'adage', 'that', 'what', 'is', 'good', 'for', 'the', 'goose', 'is', 'also', 'good', 'for', 'the', 'gander', ',', 'some', 'of', 'which', 'occasionally', 'amuses', 'but', 'none', 'of', 'which', 'amounts', 'to', 'much', 'of', 'a', 'story', '.'] type=list}" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# 将所有字母转为小写, 并所有句子变成单词序列\n", + "dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='sentence')\n", + "dataset.apply(lambda x: x['sentence'].split(), new_field_name='words', is_input=True)\n", + "dataset[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str,\n", + "'sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'words': [4, 1, 6, 1, 1, 2, 1, 11, 153, 10, 28, 17, 2, 1, 10, 1, 28, 17, 2, 1, 5, 154, 6, 149, 1, 1, 23, 1, 6, 149, 1, 8, 30, 6, 4, 35, 3] type=list}" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Vocabulary\n", + "\n", + "# 使用Vocabulary类统计单词，并将单词序列转化为数字序列\n", + "vocab = Vocabulary(min_freq=2).from_dataset(dataset, field_name='words')\n", + "vocab.index_dataset(dataset, field_name='words',new_field_name='words')\n", + "dataset[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str,\n", + "'sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'words': [4, 1, 6, 1, 1, 2, 1, 11, 153, 10, 28, 17, 2, 1, 10, 1, 28, 17, 2, 1, 5, 154, 6, 149, 1, 1, 23, 1, 6, 149, 1, 8, 30, 6, 4, 35, 3] type=list,\n", + "'target': 1 type=int}" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# 将label转为整数，并设置为 target\n", + "dataset.apply(lambda x: int(x['label']), new_field_name='target', is_target=True)\n", + "dataset[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "CNNText(\n", + " (embed): Embedding(\n", + " 177, 50\n", + " (dropout): Dropout(p=0.0)\n", + " )\n", + " (conv_pool): ConvMaxpool(\n", + " (convs): ModuleList(\n", + " (0): Conv1d(50, 3, kernel_size=(3,), stride=(1,), padding=(2,))\n", + " (1): Conv1d(50, 4, kernel_size=(4,), stride=(1,), padding=(2,))\n", + " (2): Conv1d(50, 5, kernel_size=(5,), stride=(1,), padding=(2,))\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1)\n", + " (fc): Linear(in_features=12, out_features=5, bias=True)\n", + ")" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP.models import CNNText\n", + "model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1)\n", + "model" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(62, 15)" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# 分割训练集/验证集\n", + "train_data, dev_data = dataset.split(0.2)\n", + "len(train_data), len(dev_data)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input fields after batch(if batch size is 2):\n", + "\twords: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 26]) \n", + "target fields after batch(if batch size is 2):\n", + "\ttarget: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "\n", + "training epochs started 2019-05-09-10-59-39\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=20), HTML(value='')), layout=Layout(display='…" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.333333\n", + "\n", + "Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.533333\n", + "\n", + "Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.533333\n", + "\n", + "Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.533333\n", + "\n", + "Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.6\n", + "\n", + "Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.8\n", + "\n", + "Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.8\n", + "\n", + "Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.733333\n", + "\n", + "Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.733333\n", + "\n", + "Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.733333\n", + "\n", + "\n", + "In Epoch:6/Step:12, got best dev performance:AccuracyMetric: acc=0.8\n", + "Reloaded the best model.\n" + ] + }, + { + "data": { + "text/plain": [ + "{'best_eval': {'AccuracyMetric': {'acc': 0.8}},\n", + " 'best_epoch': 6,\n", + " 'best_step': 12,\n", + " 'seconds': 0.22}" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Trainer, CrossEntropyLoss, AccuracyMetric\n", + "\n", + "# 定义trainer并进行训练\n", + "trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data,\n", + " loss=CrossEntropyLoss(), metrics=AccuracyMetric())\n", + "trainer.train()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.7" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/tutorials/tutorial_1.ipynb b/tutorials/tutorial_1.ipynb new file mode 100644 index 00000000..db302238 --- /dev/null +++ b/tutorials/tutorial_1.ipynb @@ -0,0 +1,831 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "# 详细指南" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 数据读入" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str}" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP.io import CSVLoader\n", + "\n", + "loader = CSVLoader(headers=('raw_sentence', 'label'), sep='\\t')\n", + "dataset = loader.load(\"./sample_data/tutorial_sample_dataset.csv\")\n", + "dataset[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Instance表示一个样本，由一个或多个field（域，属性，特征）组成，每个field有名字和值。\n", + "\n", + "在初始化Instance时即可定义它包含的域，使用 \"field_name=field_value\"的写法。" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': fake data type=str,\n", + "'label': 0 type=str}" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Instance\n", + "\n", + "dataset.append(Instance(raw_sentence='fake data', label='0'))\n", + "dataset[-1]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 数据处理" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str,\n", + "'sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'words': [4, 1, 6, 1, 1, 2, 1, 11, 153, 10, 28, 17, 2, 1, 10, 1, 28, 17, 2, 1, 5, 154, 6, 149, 1, 1, 23, 1, 6, 149, 1, 8, 30, 6, 4, 35, 3] type=list,\n", + "'target': 1 type=int}" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Vocabulary\n", + "\n", + "# 将所有字母转为小写, 并所有句子变成单词序列\n", + "dataset.apply(lambda x: x['raw_sentence'].lower(), new_field_name='sentence')\n", + "dataset.apply_field(lambda x: x.split(), field_name='sentence', new_field_name='words')\n", + "\n", + "# 使用Vocabulary类统计单词，并将单词序列转化为数字序列\n", + "vocab = Vocabulary(min_freq=2).from_dataset(dataset, field_name='words')\n", + "vocab.index_dataset(dataset, field_name='words',new_field_name='words')\n", + "\n", + "# 将label转为整数\n", + "dataset.apply(lambda x: int(x['label']), new_field_name='target')\n", + "dataset[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'raw_sentence': A series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'label': 1 type=str,\n", + "'sentence': a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . type=str,\n", + "'words': [4, 1, 6, 1, 1, 2, 1, 11, 153, 10, 28, 17, 2, 1, 10, 1, 28, 17, 2, 1, 5, 154, 6, 149, 1, 1, 23, 1, 6, 149, 1, 8, 30, 6, 4, 35, 3] type=list,\n", + "'target': 1 type=int,\n", + "'seq_len': 37 type=int}\n" + ] + } + ], + "source": [ + "# 增加长度信息\n", + "dataset.apply_field(lambda x: len(x), field_name='words', new_field_name='seq_len')\n", + "print(dataset[0])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用内置模块CNNText\n", + "设置为符合内置模块的名称" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "CNNText(\n", + " (embed): Embedding(\n", + " 177, 50\n", + " (dropout): Dropout(p=0.0)\n", + " )\n", + " (conv_pool): ConvMaxpool(\n", + " (convs): ModuleList(\n", + " (0): Conv1d(50, 3, kernel_size=(3,), stride=(1,), padding=(2,))\n", + " (1): Conv1d(50, 4, kernel_size=(4,), stride=(1,), padding=(2,))\n", + " (2): Conv1d(50, 5, kernel_size=(5,), stride=(1,), padding=(2,))\n", + " )\n", + " )\n", + " (dropout): Dropout(p=0.1)\n", + " (fc): Linear(in_features=12, out_features=5, bias=True)\n", + ")" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP.models import CNNText\n", + "\n", + "model_cnn = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1)\n", + "model_cnn" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "我们在使用内置模块的时候，还应该使用应该注意把 field 设定成符合内置模型输入输出的名字。" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "words\n", + "seq_len\n", + "target\n" + ] + } + ], + "source": [ + "from fastNLP import Const\n", + "\n", + "dataset.rename_field('words', Const.INPUT)\n", + "dataset.rename_field('seq_len', Const.INPUT_LEN)\n", + "dataset.rename_field('target', Const.TARGET)\n", + "\n", + "dataset.set_input(Const.INPUT, Const.INPUT_LEN)\n", + "dataset.set_target(Const.TARGET)\n", + "\n", + "print(Const.INPUT)\n", + "print(Const.INPUT_LEN)\n", + "print(Const.TARGET)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 分割训练集/验证集/测试集" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "(64, 7, 7)" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_dev_data, test_data = dataset.split(0.1)\n", + "train_data, dev_data = train_dev_data.split(0.1)\n", + "len(train_data), len(dev_data), len(test_data)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 训练(model_cnn)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### loss\n", + "训练模型需要提供一个损失函数\n", + "\n", + "下面提供了一个在分类问题中常用的交叉熵损失。注意它的**初始化参数**。\n", + "\n", + "pred参数对应的是模型的forward返回的dict的一个key的名字，这里是\"output\"。\n", + "\n", + "target参数对应的是dataset作为标签的field的名字，这里是\"label_seq\"。" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import CrossEntropyLoss\n", + "\n", + "# loss = CrossEntropyLoss()\n", + "# 等价于\n", + "loss = CrossEntropyLoss(pred=Const.OUTPUT, target=Const.TARGET)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Metric\n", + "定义评价指标\n", + "\n", + "这里使用准确率。参数的“命名规则”跟上面类似。\n", + "\n", + "pred参数对应的是模型的predict方法返回的dict的一个key的名字，这里是\"predict\"。\n", + "\n", + "target参数对应的是dataset作为标签的field的名字，这里是\"label_seq\"。" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "from fastNLP import AccuracyMetric\n", + "\n", + "# metrics=AccuracyMetric()\n", + "# 等价于\n", + "metrics=AccuracyMetric(pred=Const.OUTPUT, target=Const.TARGET)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input fields after batch(if batch size is 2):\n", + "\twords: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 16]) \n", + "\tseq_len: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "target fields after batch(if batch size is 2):\n", + "\ttarget: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "\n", + "training epochs started 2019-05-12-21-38-34\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=20), HTML(value='')), layout=Layout(display='…" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.285714\n", + "\n", + "Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.428571\n", + "\n", + "Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.428571\n", + "\n", + "Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.428571\n", + "\n", + "Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.428571\n", + "\n", + "Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.428571\n", + "\n", + "Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.428571\n", + "\n", + "Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.857143\n", + "\n", + "\n", + "In Epoch:8/Step:16, got best dev performance:AccuracyMetric: acc=0.857143\n", + "Reloaded the best model.\n" + ] + }, + { + "data": { + "text/plain": [ + "{'best_eval': {'AccuracyMetric': {'acc': 0.857143}},\n", + " 'best_epoch': 8,\n", + " 'best_step': 16,\n", + " 'seconds': 0.21}" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Trainer\n", + "\n", + "trainer = Trainer(model=model_cnn, train_data=train_data, dev_data=dev_data, loss=loss, metrics=metrics)\n", + "trainer.train()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 测试(model_cnn)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[tester] \n", + "AccuracyMetric: acc=0.857143\n" + ] + }, + { + "data": { + "text/plain": [ + "{'AccuracyMetric': {'acc': 0.857143}}" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Tester\n", + "\n", + "tester = Tester(test_data, model_cnn, metrics=AccuracyMetric())\n", + "tester.test()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 编写自己的模型\n", + "\n", + "完全支持 pytorch 的模型，与 pytorch 唯一不同的是返回结果是一个字典，字典中至少需要包含 \"pred\" 这个字段" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "\n", + "class LSTMText(nn.Module):\n", + " def __init__(self, vocab_size, embedding_dim, output_dim, hidden_dim=64, num_layers=2, dropout=0.5):\n", + " super().__init__()\n", + "\n", + " self.embedding = nn.Embedding(vocab_size, embedding_dim)\n", + " self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=num_layers, bidirectional=True, dropout=dropout)\n", + " self.fc = nn.Linear(hidden_dim * 2, output_dim)\n", + " self.dropout = nn.Dropout(dropout)\n", + "\n", + " def forward(self, words):\n", + " # (input) words : (batch_size, seq_len)\n", + " words = words.permute(1,0)\n", + " # words : (seq_len, batch_size)\n", + "\n", + " embedded = self.dropout(self.embedding(words))\n", + " # embedded : (seq_len, batch_size, embedding_dim)\n", + " output, (hidden, cell) = self.lstm(embedded)\n", + " # output: (seq_len, batch_size, hidden_dim * 2)\n", + " # hidden: (num_layers * 2, batch_size, hidden_dim)\n", + " # cell: (num_layers * 2, batch_size, hidden_dim)\n", + "\n", + " hidden = torch.cat((hidden[-2, :, :], hidden[-1, :, :]), dim=1)\n", + " hidden = self.dropout(hidden)\n", + " # hidden: (batch_size, hidden_dim * 2)\n", + "\n", + " pred = self.fc(hidden.squeeze(0))\n", + " # result: (batch_size, output_dim)\n", + " return {\"pred\":pred}" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input fields after batch(if batch size is 2):\n", + "\twords: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 16]) \n", + "\tseq_len: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "target fields after batch(if batch size is 2):\n", + "\ttarget: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "\n", + "training epochs started 2019-05-12-21-38-36\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=20), HTML(value='')), layout=Layout(display='…" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.714286\n", + "\n", + "Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.857143\n", + "\n", + "\n", + "In Epoch:6/Step:12, got best dev performance:AccuracyMetric: acc=0.857143\n", + "Reloaded the best model.\n" + ] + }, + { + "data": { + "text/plain": [ + "{'best_eval': {'AccuracyMetric': {'acc': 0.857143}},\n", + " 'best_epoch': 6,\n", + " 'best_step': 12,\n", + " 'seconds': 2.15}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "model_lstm = LSTMText(len(vocab),50,5)\n", + "trainer = Trainer(model=model_lstm, train_data=train_data, dev_data=dev_data, loss=loss, metrics=metrics)\n", + "trainer.train()" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[tester] \n", + "AccuracyMetric: acc=0.857143\n" + ] + }, + { + "data": { + "text/plain": [ + "{'AccuracyMetric': {'acc': 0.857143}}" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tester = Tester(test_data, model_lstm, metrics=AccuracyMetric())\n", + "tester.test()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 使用 Batch编写自己的训练过程" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 0 Avg Loss: 3.11 18ms\n", + "Epoch 1 Avg Loss: 2.88 30ms\n", + "Epoch 2 Avg Loss: 2.69 42ms\n", + "Epoch 3 Avg Loss: 2.47 54ms\n", + "Epoch 4 Avg Loss: 2.38 67ms\n", + "Epoch 5 Avg Loss: 2.10 78ms\n", + "Epoch 6 Avg Loss: 2.06 91ms\n", + "Epoch 7 Avg Loss: 1.92 103ms\n", + "Epoch 8 Avg Loss: 1.91 114ms\n", + "Epoch 9 Avg Loss: 1.76 126ms\n", + "[tester] \n", + "AccuracyMetric: acc=0.571429\n" + ] + }, + { + "data": { + "text/plain": [ + "{'AccuracyMetric': {'acc': 0.571429}}" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import BucketSampler\n", + "from fastNLP import Batch\n", + "import torch\n", + "import time\n", + "\n", + "model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1)\n", + "\n", + "def train(epoch, data):\n", + " optim = torch.optim.Adam(model.parameters(), lr=0.001)\n", + " lossfunc = torch.nn.CrossEntropyLoss()\n", + " batch_size = 32\n", + "\n", + " # 定义一个Batch，传入DataSet，规定batch_size和去batch的规则。\n", + " # 顺序（Sequential），随机（Random），相似长度组成一个batch（Bucket）\n", + " train_sampler = BucketSampler(batch_size=batch_size, seq_len_field_name='seq_len')\n", + " train_batch = Batch(batch_size=batch_size, dataset=data, sampler=train_sampler)\n", + " \n", + " start_time = time.time()\n", + " for i in range(epoch):\n", + " loss_list = []\n", + " for batch_x, batch_y in train_batch:\n", + " optim.zero_grad()\n", + " output = model(batch_x['words'])\n", + " loss = lossfunc(output['pred'], batch_y['target'])\n", + " loss.backward()\n", + " optim.step()\n", + " loss_list.append(loss.item())\n", + " print('Epoch {:d} Avg Loss: {:.2f}'.format(i, sum(loss_list) / len(loss_list)),end=\" \")\n", + " print('{:d}ms'.format(round((time.time()-start_time)*1000)))\n", + " loss_list.clear()\n", + " \n", + "train(10, train_data)\n", + "tester = Tester(test_data, model, metrics=AccuracyMetric())\n", + "tester.test()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 使用 Callback 实现自己想要的效果" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input fields after batch(if batch size is 2):\n", + "\twords: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2, 16]) \n", + "\tseq_len: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "target fields after batch(if batch size is 2):\n", + "\ttarget: (1)type:torch.Tensor (2)dtype:torch.int64, (3)shape:torch.Size([2]) \n", + "\n", + "training epochs started 2019-05-12-21-38-40\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "HBox(children=(IntProgress(value=0, layout=Layout(flex='2'), max=20), HTML(value='')), layout=Layout(display='…" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluation at Epoch 1/10. Step:2/20. AccuracyMetric: acc=0.285714\n", + "\n", + "Sum Time: 51ms\n", + "\n", + "\n", + "Evaluation at Epoch 2/10. Step:4/20. AccuracyMetric: acc=0.285714\n", + "\n", + "Sum Time: 69ms\n", + "\n", + "\n", + "Evaluation at Epoch 3/10. Step:6/20. AccuracyMetric: acc=0.285714\n", + "\n", + "Sum Time: 91ms\n", + "\n", + "\n", + "Evaluation at Epoch 4/10. Step:8/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Sum Time: 107ms\n", + "\n", + "\n", + "Evaluation at Epoch 5/10. Step:10/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Sum Time: 125ms\n", + "\n", + "\n", + "Evaluation at Epoch 6/10. Step:12/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Sum Time: 142ms\n", + "\n", + "\n", + "Evaluation at Epoch 7/10. Step:14/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Sum Time: 158ms\n", + "\n", + "\n", + "Evaluation at Epoch 8/10. Step:16/20. AccuracyMetric: acc=0.571429\n", + "\n", + "Sum Time: 176ms\n", + "\n", + "\n", + "Evaluation at Epoch 9/10. Step:18/20. AccuracyMetric: acc=0.714286\n", + "\n", + "Sum Time: 193ms\n", + "\n", + "\n", + "Evaluation at Epoch 10/10. Step:20/20. AccuracyMetric: acc=0.857143\n", + "\n", + "Sum Time: 212ms\n", + "\n", + "\n", + "\n", + "In Epoch:10/Step:20, got best dev performance:AccuracyMetric: acc=0.857143\n", + "Reloaded the best model.\n" + ] + }, + { + "data": { + "text/plain": [ + "{'best_eval': {'AccuracyMetric': {'acc': 0.857143}},\n", + " 'best_epoch': 10,\n", + " 'best_step': 20,\n", + " 'seconds': 0.2}" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from fastNLP import Callback\n", + "\n", + "start_time = time.time()\n", + "\n", + "class MyCallback(Callback):\n", + " def on_epoch_end(self):\n", + " print('Sum Time: {:d}ms\\n\\n'.format(round((time.time()-start_time)*1000)))\n", + " \n", + "\n", + "model = CNNText((len(vocab),50), num_classes=5, padding=2, dropout=0.1)\n", + "trainer = Trainer(model=model, train_data=train_data, dev_data=dev_data,\n", + " loss=CrossEntropyLoss(), metrics=AccuracyMetric(), callbacks=[MyCallback()])\n", + "trainer.train()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.7" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/tutorials/tutorial_for_developer.md b/tutorials/tutorial_for_developer.md deleted file mode 100644 index b2ec0b98..00000000 --- a/tutorials/tutorial_for_developer.md +++ /dev/null @@ -1,283 +0,0 @@ -# fastNLP开发者指南 -#### 本教程涉及以下类： -- DataSet -- Sampler -- Batch -- Model -- Loss -- Metric -- Trainer -- Tester - -#### DataSet: 用于承载数据。 -1. DataSet里面每个元素只能是以下的三类`np.float64`, `np.int64`, `np.str`。如果传入的数据是`int`则被转换为`np.int64`, `float`被转为`np.float64`。 -2. DataSet可以将field设置为input或者target。其中被设置为input的field会被传递给Model.forward, 这个过程中我们是通过键匹配完成传递的。举例来说，假设DataSet中有'x1', 'x2', 'x3'被设置为了input，而 - - 函数是Model.forward(self, x1, x3), 那么DataSet中'x1', 'x3'会被传递给forward函数。多余的'x2'会被忽略 - - 函数是Model.forward(self, x1, x4), 这里多需要了一个'x4', 但是DataSet的input field中没有这个field，会报错。 - - 函数是Model.forward(self, x1, **kwargs), 会把'x1', 'x2', 'x3'都传入。但如果是Model.forward(self, x4, **kwargs)就会发生报错，因为没有'x4'。 -3. 对于设置为target的field的名称，我们建议取名为'target'（如果只有一个需要predict的值），但是不强制。后面会讲为什么target可以不强制。 -DataSet应该是不需要单独再开发的，如果有不能满足的场景，请在开发群提出或者github提交issue。 - -#### Sampler: 给定一个DataSet，返回一个序号的list，Batch按照这个list输出数据。 -Sampler需要继承fastNLP.core.sampler.BaseSampler -```python -class BaseSampler(object): - """The base class of all samplers. - - Sub-classes must implement the __call__ method. - __call__ takes a DataSet object and returns a list of int - the sampling indices. - """ -def __call__(self, *args, **kwargs): - raise NotImplementedError - -# 子类需要复写__call__方法。这个函数只能有一个必选参数, 且必须是DataSet类别， 否则Trainer没法调 -class SonSampler(BaseSampler): - def __init__(self, xxx): - # 可以实现init也不可以不实现。 - pass - def __call__(self, data_set): - pass -``` - -#### Batch: 将DataSet中设置为input和target的field取出来构成batch_x, batch_y -并且根据情况(主要根据数据类型能不能转为Tensor)将数据转换为pytorch的Tensor。batch中sample的取出顺序是由Sampler决定的。 -Sampler是传入一个DataSet，返回一个与DataSet等长的序号list，Batch一次会取出batch_size个sample(最后一个batch可能数量不足batch_size个)。 -举例： -1. SequentialSampler是顺序采样 - - 假设传入的DataSet长度是100, SequentialSampler返回的序号list就是[0, 1, ...,98, 99]. batch_size如果被设置为4，那么第一个batch所获取的instance就是[0, 1, 2, 3]这四个instance. 第二个batch所获取instace就是[4, 5, 6, 7], ...直到采完所有的sample。 -2. RandomSampler是随机采样 - - 假设传入的DataSet长度是100, RandomSampler返回的序号list可能是[0, 99, 20, 5, 3, 1, ...]. 依次按照batch_size的大小取出sample。 - -Batch应该不需要继承与开发，如果你有特殊需求请在开发群里提出。 - -#### Model：用户自定的Model -必须是nn.Module的子类 -1. 必须实现forward方法，并且forward方法不能出现*arg这种参数. 例如 - ```python - def forward(self, word_seq, *args): #这是不允许的. - # ... - pass - ``` - 返回值必须是dict的 - ```python - def forward(self, word_seq, seq_lens): - xxx = "xxx" - return {'pred': xxx} #return的值必须是dict的。里面的预测的key推荐使用pred，但是不做强制限制。输出元素数目不限。 - ``` -2. 如果实现了predict方法，在做evaluation的时候将调用predict方法而不是forward。如果没有predict方法，则在evaluation时调用forward方法。predict方法也不能使用*args这种参数形式，同时结果也必须返回一个dict，同样推荐key为'pred'。 - -#### Loss: 根据model.forward()返回的prediction(是一个dict)和batch_y计算相应的loss -1. 先介绍"键映射"。 如在DataSet, Model一节所看见的那样，fastNLP并不限制Model.forward()的返回值，也不限制DataSet中target field的key。计算的loss的时候，怎么才能知道从哪里取值呢？ -这里以CrossEntropyLoss为例，一般情况下, 计算CrossEntropy需要prediction和target两个值。而在CrossEntropyLoss初始化时可以传入两个参数(pred=None, target=None), 这两个参数接受的类型是str，假设(pred='output', target='label')，那么CrossEntropyLoss会使用'output'这个key在forward的output与batch_y中寻找值;'label'也是在forward的output与batch_y中寻找值。注意这里pred或target的来源并不一定非要来自于model.forward与batch_y，也可以只来自于forward的结果。 -2. 如何创建一个自己的loss - - 使用fastNLP.LossInForward, 在model.forward()的结果中包含一个为loss的key。 - - trainer中使用loss(假设loss=CrossEntropyLoss())的时候其实是 - los = loss(prediction, batch_y)，即直接调用的是`loss.__call__()`方法，但是CrossEntropyLoss里面并没有自己实现`__call__`方法，这是因为`__call__`在LossBase中实现了。所有的loss必须继承fastNLP.core.loss.LossBase, 下面先说一下LossBase的几个方法，见下一节。 -3. 尽量不要复写`__call__()`, `_init_param_map()`方法。 - -```python -class LossBase(): - def __init__(self): - self.param_map = {} # 一般情况下也不需要自己创建。调用_init_param_map()更好 - self._checked = False # 这个参数可以忽略 - - def _init_param_map(self, key_map=None, **kwargs): - # 这个函数是用于注册Loss的“键映射”，有两种传值方法， - # 第一种是通过key_map传入dict，取值是用value到forward和batch_y取 - # key_map = {'pred': 'output', 'target': 'label'} - # 第二种是自己写 - # _init_param_map(pred='output', target='label') - # 为什么会提供这么一个方法？通过调用这个方法会自动注册param_map，并会做一些检查，防止出现传入的key其实并不是get_loss - # 的一个参数。注意传入这个方法的参数必须都是需要做键映射的内容，其它loss参数不要传入。如果传入(pred=None, target=None) - # 则__call__()会到pred_dict与target_dict去寻找key为'pred'和'target'的值。 - # 但这个参数不是必须要调用的。 - - def __call__(self, pred_dict, target_dict, check=False): # check=False忽略这个参数，之后应该会被删除的 - # 这个函数主要会做一些check的工作，比如pred_dict与target_dict中是否包含了计算loss所必须的key等。检查通过，则调用get_loss - # 方法。 - fast_param = self._fast_param_map(predict_dict, target_dict): - if fast_param: - return self.get_loss(**fast_param) - # 如果没有fast_param则通过匹配参数然后调用get_loss完成 - xxxx - return loss # 返回为Tensor的loss - def _fast_param_map(self, pred_dict, target_dict): - # 这是一种快速计算loss的机制，因为在很多情况下其实都不需要通过"键映射"，比如计算loss时，pred_dict只有一个元素， - # target_dict也只有一个元素，那么无歧义地就可以把预测值与实际值用于计算loss, 基类判断了这种情况(可能还有其它无歧义的情况)。 - # 即_fast_param_map成功的话，就不需要使用键映射，这样即使在没有传递或者传递错误"键映射"的情况也可以直接计算loss。 - # 返回值是一个dict, 如果匹配成功，应该返回类似{'pred':value, 'target': value}的结果；如果dict为空则说明匹配失败， - # __call__方法会继续执行。 - - def get_loss(self, *args, **kwargs): - # 这个是一定需要实现的，计算loss的地方。 - # (1) get_loss中一定不能包含*arg这种参数形式。 - # (2) 如果包含**kwargs这种参数，这会将pred_dict与target_dict中所有参数传入。但是建议不要用这个参数 - raise NotImplementedError - -# 下面使用L1Loss举例 -class L1Loss(LossBase): # 继承LossBase - # 初始化需要映射的值，这里需要映射的值'pred', 'target'必须与get_loss需要参数名是对应的 - def __init__(self, pred=None, target=None): - super(L1Loss, self).__init__() - # 这里传入_init_param_map以使得pred和target被正确注册，但这一步不是必须的, 建议调用。传入_init_param_map的是用于 - # “键映射"的键值对。假设初始化__init__(pred=None, target=None, threshold=0.1)中threshold是用于控制loss计算的，则 - # 不要将threshold传入_init_param_map. - self._init_param_map(pred=pred, target=target) - - def get_loss(self, pred, target): - # 这里'pred', 'target'必须和初始化的映射是一致的。 - return F.l1_loss(input=pred, target=target) #直接返回一个loss即可 -``` - -### Metric: 根据Model.forward()或者Model.predict()的结果计算metric -metric的设计和loss的设计类似。都是传入pred_dict与target_dict进行计算。但是metric的pred_dict来源可能是Model.forward的返回值， 也可能是Model.predict(如果Model具有predict方法则会调用predict方法)的返回值，下面统一用pred_dict代替。 -1. 这里的"键映射"与loss的"键映射"是类似的。举例来说，若Metric(pred='output', target='label')，则使用'output'到pred_dict和target_dict中寻找pred, 用'label'寻找target。 -2. 如何创建一个自己的Metric方法 -Metric与loss的计算不同在于，Metric的计算有两个步骤。 - - **每个batch的输出**都会调用Metric的``__call__(pred_dict, target_dict)``方法，而``__call__``方法会调用evaluate()(需要实现)方法。 - - 在所有batch传入之后，调用Metric的get_metric()方法得到最终的metric值。 - - 所以Metric在调用evaluate方法时，根据拿到的数据: pred_dict与batch_y, 改变自己的状态(比如累加正确的次数，总的sample数等)。在调用get_metric()的时候给出一个最终计算结果。 - 所有的Metric必须继承自fastNLP.core.metrics.MetricBase. 例子见下一个cell -3. 尽量不要复写``__call__()``,``_init_param_map()``方法。 - -```python -class MetricBase: - def __init__(self): - self.param_map = {} # 一般情况下也不需要自己创建。调用_init_param_map()更好 - self._checked = False # 这个参数可以忽略 - - def _init_param_map(self, key_map=None, **kwargs): - # 这个函数是用于注册Metric的“键映射”，有两种传值方法， - # 第一种是通过key_map传入dict，取值是用value到forward和batch_y取 - # key_map = {'pred': 'output', 'target': 'label'} - # 第二种是自己写(建议使用改种方式) - # _init_param_map(pred='output', target='label') - # 为什么会提供这么一个方法？通过调用这个方法会自动注册param_map，并会做一些检查，防止出现传入的key其实并不是evaluate() - # 的一个参数。注意传入这个方法的参数必须都是需要做键映射的内容，其它evaluate参数不要传入。如果传入(pred=None, target=None) - # 则__call__()会到pred_dict与target_dict去寻找key为'pred'和'target'的值。 - # 但这个参数不是必须要调用的。 - pass - - def __call__(self, pred_dict, target_dict, check=False): # check=False忽略这个参数，之后应该会被删除的 - # 这个函数主要会做一些check的工作，比如pred_dict与target_dict中是否包含了计算evaluate所必须的key等。检查通过，则调用 - # evaluate方法。 - fast_param = self._fast_param_map(predict_dict, target_dict): - if fast_param: - return self.evaluate(**fast_param) - # 如果没有fast_param则通过匹配参数然后调用get_loss完成 - # xxxx - - def _fast_param_map(self, pred_dict, target_dict): - # 这是一种快速计算loss的机制，因为在很多情况下其实都不需要通过"键映射"，比如evaluate时，pred_dict只有一个元素， - # target_dict也只有一个元素，那么无歧义地就可以把预测值与实际值用于计算metric, 基类判断了这种情况(可能还有其它无歧义的 - # 情况)。即_fast_param_map成功的话，就不需要使用键映射，这样即使在没有传递或者传递错误"键映射"的情况也可以直接计算metric。 - # 返回值是一个dict, 如果匹配成功，应该返回类似{'pred':value, 'target': value}的结果；如果dict为空则说明匹配失败， - # __call__方法会继续尝试匹配。 - pass - - def evaluate(self, *args, **kwargs): - # 这个是一定需要实现的，累加metric状态 - # (1) evaluate()中一定不能包含*arg这种参数形式。 - # (2) 如果包含**kwargs这种参数，这会将pred_dict与target_dict中所有参数传入。但是建议不要用这个参数 - raise NotImplementedError - - def get_metric(self, reset=True): - # 这是一定需要实现的，获取最终的metric。返回值必须是一个dict。会在所有batch传入之后调用 - raise NotImplementedError - -# 下面使用AccuracyMetric举例 -class AccuracyMetric(MetricBase): # MetricBase - # 初始化需要映射的值，这里需要映射的值'pred', 'target'必须与evaluate()需要参数名是对应的 - def __init__(self, pred=None, target=None): - super(AccuracyMetric, self).__init__() - # 这里传入_init_param_map以使得pred和target被正确注册，但这一步不是必须的, 建议调用。传入_init_param_map的是用于 - # “键映射"的键值对。假设初始化__init__(pred=None, target=None, threshold=0.1)中threshold是用于控制loss计算的，则 - # 不要将threshold传入_init_param_map. - self._init_param_map(pred=pred, target=target) - - self.total = 0 # 用于累加一共有多少sample - self.corr = 0 # 用于累加一共有多少正确的sample - - def evaluate(self, pred, target): - # 对pred和target做一些基本的判断或者预处理等 - if pred.size()==target.size() and len(pred.size())=1: #如果pred已经做了argmax - pass - elif len(pred.size())==2 and len(target.size())==1: # pred还没有进行argmax - pred = pred.argmax(dim=1) - else: - raise ValueError("The shape of pred and target should be ((B, n_classes), (B, )) or (" - "(B,),(B,)).") - assert pred.size(0)==target.size(0), "Mismatch batch size." - # 进行相应的累加 - self.total += pred.size(0) - self.corr += torch.sum(torch.eq(pred, target).float()).item() - - def get_metric(self, reset=True): - # reset用于指示是否清空累加信息。默认为True - # 这个函数需要返回dict，可以包含多个metric。 - metric = {} - metric['acc'] = self.corr/self.total - if reset: - self.total = 0 - self.corr = 0 - return metric -``` - -#### Tester: 用于做evaluation，应该不需要更改 -重要的初始化参数有data, model, metric；比较重要的function是test()。 - -test中的运行过程 -``` -predict_func = 如果有model.predict则为model.predict, 否则是model.forward -for batch_x, batch_y in batch: -# (1) 同步数据与model -# (2) 根据predict_func的参数从batch_x中取出数据传入到predict_func中，得到结果pred_dict -# (3) 调用metric(pred_dict, batch_y -# (4) 当所有batch都运行完毕，会调用metric的get_metric方法，并且以返回的值作为evaluation的结果 -metric.get_metric() -``` - -#### Trainer: 对训练过程的封装。 -里面比较重要的function是train() -train()中的运行过程 -``` -(1) 创建batch - batch = Batch(dataset, batch_size, sampler=sampler) - for batch_x, batch_y in batch: - # ... - batch_x，batch_y都是dict。batch_x是DataSet中被设置为input的field；batch_y是DataSet中被设置为target的field。 - 两个dict中的key就是DataSet中的key，value会根据情况做好padding的tensor。 -(2)会将batch_x, batch_y中tensor移动到model所在的device -(3)根据model.forward的参数列表, 从batch_x中取出需要传递给forward的数据。 -(4)获取model.forward的输出结果pred_dict，并与batch_y一起传递给loss函数, 求得loss -(5)对loss进行反向梯度并更新参数 -(6) 如果有验证集，则需要做验证 - tester = Tester(model, dev_data，metric) - eval_results = tester.test() -(7) 如果eval_results是当前的最佳结果，则保存模型。 -``` - -#### 其他 -Trainer中还提供了"预跑"的功能。该功能通过check_code_level管理，如果check_code_level为-1，则不进行"预跑"。 - -check_code_level=0，1，2代表不同的提醒级别。 -目前不同提醒级别对应的是对DataSet中设置为input或target但又没有使用的field的提醒级别。 -0是忽略(默认)；1是会warning发生了未使用field的情况；2是出现了unused会直接报错并退出运行 - -"预跑"的主要目的有两个: -- 防止train完了之后进行evaluation的时候出现错误。之前的train就白费了 -- 由于存在"键映射"，直接运行导致的报错可能不太容易debug，通过"预跑"过程的报错会有一些debug提示 - -"预跑"会进行以下的操作： -- 使用很小的batch_size, 检查batch_x中是否包含Model.forward所需要的参数。只会运行两个循环。 -- 将Model.foward的输出pred_dict与batch_y输入到loss中， 并尝试backward. 不会更新参数，而且grad会被清零 - 如果传入了dev_data，还将进行metric的测试 -- 创建Tester，并传入少量数据，检测是否可以正常运行 - -"预跑"操作是在Trainer初始化的时候执行的。 - -正常情况下，应该不需要改动"预跑"的代码。但如果你遇到bug或者有什么好的建议，欢迎在开发群或者github提交issue。 - -