@@ -19,9 +19,9 @@ | |||
</div> | |||
# ABL kit: A Python Toolkit for Abductive Learning | |||
# ABLkit: A Python Toolkit for Abductive Learning | |||
**ABL kit** is an efficient Python toolkit for **Abductive Learning (ABL)**. | |||
**ABLkit** is an efficient Python toolkit for **Abductive Learning (ABL)**. | |||
ABL is a novel paradigm that integrates machine learning and | |||
logical reasoning in a unified framework. It is suitable for tasks | |||
where both data and (logical) domain knowledge are available. | |||
@@ -30,25 +30,25 @@ where both data and (logical) domain knowledge are available. | |||
<img src="./docs/_static/img/ABL.png" alt="Abductive Learning" style="width: 80%;"/> | |||
</p> | |||
Key Features of ABL kit: | |||
Key Features of ABLkit: | |||
- **Great Flexibility**: Adaptable to various machine learning modules and logical reasoning components. | |||
- **User-Friendly**: Provide data, model, and KB, and get started with just a few lines of code. | |||
- **High-Performance**: Optimization for high accuracy and fast training speed. | |||
ABL kit encapsulates advanced ABL techniques, providing users with | |||
ABLkit encapsulates advanced ABL techniques, providing users with | |||
an efficient and convenient toolkit to develop dual-driven ABL systems, | |||
which leverage the power of both data and knowledge. | |||
<p align="center"> | |||
<img src="./docs/_static/img/ABLkit.png" alt="ABL kit" style="width: 80%;"/> | |||
<img src="./docs/_static/img/ABLkit.png" alt="ABLkit" style="width: 80%;"/> | |||
</p> | |||
## Installation | |||
### Install from PyPI | |||
The easiest way to install ABL kit is using ``pip``: | |||
The easiest way to install ABLkit is using ``pip``: | |||
```bash | |||
pip install ablkit | |||
@@ -84,7 +84,7 @@ We use the MNIST Addition task as a quick start example. In this task, pairs of | |||
<summary>Working with Data</summary> | |||
<br> | |||
ABL kit requires data in the format of `(X, gt_pseudo_label, Y)` where `X` is a list of input examples containing instances, `gt_pseudo_label` is the ground-truth label of each example in `X` and `Y` is the ground-truth reasoning result of each example in `X`. Note that `gt_pseudo_label` is only used to evaluate the machine learning model's performance but not to train it. | |||
ABLkit requires data in the format of `(X, gt_pseudo_label, Y)` where `X` is a list of input examples containing instances, `gt_pseudo_label` is the ground-truth label of each example in `X` and `Y` is the ground-truth reasoning result of each example in `X`. Note that `gt_pseudo_label` is only used to evaluate the machine learning model's performance but not to train it. | |||
In the MNIST Addition task, the data loading looks like: | |||
@@ -103,7 +103,7 @@ test_data = get_dataset(train=False) | |||
<summary>Building the Learning Part</summary> | |||
<br> | |||
Learning part is constructed by first defining a base model for machine learning. ABL kit offers considerable flexibility, supporting any base model that conforms to the scikit-learn style (which requires the implementation of fit and predict methods), or a PyTorch-based neural network (which has defined the architecture and implemented forward method). In this example, we build a simple LeNet5 network as the base model. | |||
Learning part is constructed by first defining a base model for machine learning. ABLkit offers considerable flexibility, supporting any base model that conforms to the scikit-learn style (which requires the implementation of fit and predict methods), or a PyTorch-based neural network (which has defined the architecture and implemented forward method). In this example, we build a simple LeNet5 network as the base model. | |||
```python | |||
# The 'models' module below is located in 'examples/mnist_add/' | |||
@@ -112,7 +112,7 @@ from models.nn import LeNet5 | |||
cls = LeNet5(num_classes=10) | |||
``` | |||
To facilitate uniform processing, ABL kit provides the `BasicNN` class to convert a PyTorch-based neural network into a format compatible with scikit-learn models. To construct a `BasicNN` instance, aside from the network itself, we also need to define a loss function, an optimizer, and the computing device. | |||
To facilitate uniform processing, ABLkit provides the `BasicNN` class to convert a PyTorch-based neural network into a format compatible with scikit-learn models. To construct a `BasicNN` instance, aside from the network itself, we also need to define a loss function, an optimizer, and the computing device. | |||
```python | |||
import torch | |||
@@ -167,7 +167,7 @@ reasoner = Reasoner(kb) | |||
<summary>Building Evaluation Metrics</summary> | |||
<br> | |||
ABL kit provides two basic metrics, namely `SymbolAccuracy` and `ReasoningMetric`, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the `logic_forward` results, respectively. | |||
ABLkit provides two basic metrics, namely `SymbolAccuracy` and `ReasoningMetric`, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the `logic_forward` results, respectively. | |||
```python | |||
from ablkit.data.evaluation import ReasoningMetric, SymbolAccuracy | |||
@@ -7,7 +7,7 @@ from lambdaLearn.Base.TabularMixin import TabularMixin | |||
class DataConverter: | |||
""" | |||
This class provides functionality to convert LambdaLearn data to ABL kit data. | |||
This class provides functionality to convert LambdaLearn data to ABLkit data. | |||
""" | |||
def __init__(self) -> None: | |||
@@ -21,9 +21,9 @@ IndexType = Union[str, slice, int, list, LongTypeTensor, BoolTypeTensor, np.ndar | |||
class ListData(BaseDataElement): | |||
""" | |||
Abstract Data Interface used throughout the ABL kit. | |||
Abstract Data Interface used throughout the ABLkit. | |||
``ListData`` is the underlying data structure used in the ABL kit, | |||
``ListData`` is the underlying data structure used in the ABLkit, | |||
designed to manage diverse forms of data dynamically generated throughout the | |||
Abductive Learning (ABL) framework. This includes handling raw data, predicted | |||
pseudo-labels, abduced pseudo-labels, pseudo-label indices, etc. | |||
@@ -9,7 +9,7 @@ from lambdaLearn.Base.DeepModelMixin import DeepModelMixin | |||
class ModelConverter: | |||
""" | |||
This class provides functionality to convert LambdaLearn models to ABL kit models. | |||
This class provides functionality to convert LambdaLearn models to ABLkit models. | |||
""" | |||
def __init__(self) -> None: | |||
@@ -234,7 +234,7 @@ examples. | |||
.. code:: python | |||
from ablkit.data.structures import ListData | |||
# ListData is a data structure provided by ABL kit that can be used to organize data examples | |||
# ListData is a data structure provided by ABLkit that can be used to organize data examples | |||
data_examples = ListData() | |||
# We use the first 1001st and 3001st data examples in the training set as an illustration | |||
data_examples.X = [X_1000, X_3000] | |||
@@ -203,7 +203,7 @@ examples. | |||
.. code:: python | |||
from ablkit.data.structures import ListData | |||
# ListData is a data structure provided by ABL kit that can be used to organize data examples | |||
# ListData is a data structure provided by ABLkit that can be used to organize data examples | |||
data_examples = ListData() | |||
# We use the first 100 data examples in the training set as an illustration | |||
data_examples.X = train_X[:100] | |||
@@ -84,7 +84,7 @@ Out: | |||
Next, we transform the tabular data to the format required by | |||
ABL kit, which is a tuple of (X, gt_pseudo_label, Y). In this task, | |||
ABLkit, which is a tuple of (X, gt_pseudo_label, Y). In this task, | |||
we treat the attributes as X and the targets as gt_pseudo_label (ground | |||
truth pseudo-labels). Y (reasoning results) are expected to be 0, | |||
indicating no rules are violated. | |||
@@ -9,14 +9,14 @@ | |||
Learn the Basics | |||
================ | |||
Modules in ABL kit | |||
Modules in ABLkit | |||
---------------------- | |||
ABL kit is an efficient toolkit for `Abductive Learning <../Overview/Abductive-Learning.html>`_ (ABL), | |||
ABLkit is an efficient toolkit for `Abductive Learning <../Overview/Abductive-Learning.html>`_ (ABL), | |||
a paradigm which integrates machine learning and logical reasoning in a balanced-loop. | |||
ABL kit comprises three primary parts: **Data**, **Learning**, and | |||
ABLkit comprises three primary parts: **Data**, **Learning**, and | |||
**Reasoning**, corresponding to the three pivotal components of current | |||
AI: data, models, and knowledge. Below is an overview of the ABL kit. | |||
AI: data, models, and knowledge. Below is an overview of the ABLkit. | |||
.. image:: ../_static/img/ABLkit.png | |||
@@ -50,7 +50,7 @@ from the ``BaseBridge`` class). The Bridge part synthesizes data, | |||
learning, and reasoning, facilitating the training and testing | |||
of the entire ABL framework. | |||
Use ABL kit Step by Step | |||
Use ABLkit Step by Step | |||
---------------------------- | |||
In a typical ABL process, as illustrated below, | |||
@@ -10,7 +10,7 @@ | |||
Bridge | |||
====== | |||
In this section, we will look at how to bridge learning and reasoning parts to train the model, which is the fundamental idea of Abductive Learning. ABL kit implements a set of bridge classes to achieve this. | |||
In this section, we will look at how to bridge learning and reasoning parts to train the model, which is the fundamental idea of Abductive Learning. ABLkit implements a set of bridge classes to achieve this. | |||
.. code:: python | |||
@@ -42,7 +42,7 @@ In this section, we will look at how to bridge learning and reasoning parts to t | |||
| ``test(test_data)`` | Test the model. | | |||
+---------------------------------------+----------------------------------------------------+ | |||
where ``train_data`` and ``test_data`` are both in the form of a tuple or a `ListData <../API/ablkit.data.html#structures.ListData>`_. Regardless of the form, they all need to include three components: ``X``, ``gt_pseudo_label`` and ``Y``. Since ``ListData`` is the underlying data structure used throughout the ABL kit, tuple-formed data will be firstly transformed into ``ListData`` in the ``train`` and ``test`` methods, and such ``ListData`` instances are referred to as ``data_examples``. More details can be found in `preparing datasets <Datasets.html>`_. | |||
where ``train_data`` and ``test_data`` are both in the form of a tuple or a `ListData <../API/ablkit.data.html#structures.ListData>`_. Regardless of the form, they all need to include three components: ``X``, ``gt_pseudo_label`` and ``Y``. Since ``ListData`` is the underlying data structure used throughout the ABLkit, tuple-formed data will be firstly transformed into ``ListData`` in the ``train`` and ``test`` methods, and such ``ListData`` instances are referred to as ``data_examples``. More details can be found in `preparing datasets <Datasets.html>`_. | |||
``SimpleBridge`` inherits from ``BaseBridge`` and provides a basic implementation. Besides the ``model`` and ``reasoner``, ``SimpleBridge`` has an extra initialization argument, ``metric_list``, which will be used to evaluate model performance. Its training process involves several Abductive Learning loops and each loop consists of the following five steps: | |||
@@ -10,7 +10,7 @@ | |||
Dataset & Data Structure | |||
======================== | |||
In this section, we will look at the dataset and data structure in ABL kit. | |||
In this section, we will look at the dataset and data structure in ABLkit. | |||
.. code:: python | |||
@@ -20,7 +20,7 @@ In this section, we will look at the dataset and data structure in ABL kit. | |||
Dataset | |||
------- | |||
ABL kit requires user data to be either structured as a tuple ``(X, gt_pseudo_label, Y)`` or a ``ListData`` (the underlying data structure utilized in ABL kit, cf. the next section) object with ``X``, ``gt_pseudo_label`` and ``Y`` attributes. Regardless of the chosen format, the data should encompass three essential components: | |||
ABLkit requires user data to be either structured as a tuple ``(X, gt_pseudo_label, Y)`` or a ``ListData`` (the underlying data structure utilized in ABLkit, cf. the next section) object with ``X``, ``gt_pseudo_label`` and ``Y`` attributes. Regardless of the chosen format, the data should encompass three essential components: | |||
- ``X``: List[List[Any]] | |||
@@ -62,11 +62,11 @@ where each sublist in ``X``, e.g., |data_example|, is a data example and each im | |||
Data Structure | |||
-------------- | |||
Besides the user-provided dataset, various forms of data are utilized and dynamicly generated throughout the training and testing process of ABL framework. Examples include raw data, predicted pseudo-label, abduced pseudo-label, pseudo-label indices, etc. To manage this diversity and ensure a stable, versatile interface, ABL kit employs `abstract data interfaces <../API/ablkit.data.html#structures>`_ to encapsulate different forms of data that will be used in the total learning process. | |||
Besides the user-provided dataset, various forms of data are utilized and dynamicly generated throughout the training and testing process of ABL framework. Examples include raw data, predicted pseudo-label, abduced pseudo-label, pseudo-label indices, etc. To manage this diversity and ensure a stable, versatile interface, ABLkit employs `abstract data interfaces <../API/ablkit.data.html#structures>`_ to encapsulate different forms of data that will be used in the total learning process. | |||
``ListData`` is the underlying abstract data interface utilized in ABL kit. As the fundamental data structure, ``ListData`` implements commonly used data manipulation methods and is responsible for transferring data between various components of ABL, ensuring that stages such as prediction, abductive reasoning, and training can utilize ``ListData`` as a unified input format. Before proceeding to other stages, user-provided datasets will be firstly converted into ``ListData``. | |||
``ListData`` is the underlying abstract data interface utilized in ABLkit. As the fundamental data structure, ``ListData`` implements commonly used data manipulation methods and is responsible for transferring data between various components of ABL, ensuring that stages such as prediction, abductive reasoning, and training can utilize ``ListData`` as a unified input format. Before proceeding to other stages, user-provided datasets will be firstly converted into ``ListData``. | |||
Besides providing a tuple of ``(X, gt_pseudo_label, Y)``, ABL kit also allows users to directly supply data in ``ListData`` format, which similarly requires the inclusion of these three attributes. The following code shows the basic usage of ``ListData``. More information can be found in the `API documentation <../API/ablkit.data.html#structures>`_. | |||
Besides providing a tuple of ``(X, gt_pseudo_label, Y)``, ABLkit also allows users to directly supply data in ``ListData`` format, which similarly requires the inclusion of these three attributes. The following code shows the basic usage of ``ListData``. More information can be found in the `API documentation <../API/ablkit.data.html#structures>`_. | |||
.. code-block:: python | |||
@@ -16,7 +16,7 @@ In this section, we will look at how to build evaluation metrics. | |||
from ablkit.data.evaluation import BaseMetric, SymbolAccuracy, ReasoningMetric | |||
ABL kit seperates the evaluation process from model training and testing as an independent class, ``BaseMetric``. The training and testing processes are implemented in the ``BaseBridge`` class, so metrics are used by this class and its sub-classes. After building a ``bridge`` with a list of ``BaseMetric`` instances, these metrics will be used by the ``bridge.valid`` method to evaluate the model performance during training and testing. | |||
ABLkit seperates the evaluation process from model training and testing as an independent class, ``BaseMetric``. The training and testing processes are implemented in the ``BaseBridge`` class, so metrics are used by this class and its sub-classes. After building a ``bridge`` with a list of ``BaseMetric`` instances, these metrics will be used by the ``bridge.valid`` method to evaluate the model performance during training and testing. | |||
To customize our own metrics, we need to inherit from ``BaseMetric`` and implement the ``process`` and ``compute_metrics`` methods. | |||
@@ -12,7 +12,7 @@ Learning Part | |||
In this section, we will look at how to build the learning part. | |||
In ABL kit, building the learning part involves two steps: | |||
In ABLkit, building the learning part involves two steps: | |||
1. Build a machine learning base model used to make predictions on instance-level data. | |||
2. Instantiate an ``ABLModel`` with the base model, which enables the learning part to process example-level data. | |||
@@ -76,7 +76,7 @@ Besides the necessary methods required to instantiate an ``ABLModel``, i.e., ``f | |||
Instantiating an ABLModel | |||
------------------------- | |||
Typically, base model is trained to make predictions on instance-level data, and can not directly process example-level data, which is not suitable for most neural-symbolic tasks. ABL kit provides the ``ABLModel`` to solve this problem. This class serves as a unified wrapper for all base models, which enables the learning part to train, test, and predict on example-level data. | |||
Typically, base model is trained to make predictions on instance-level data, and can not directly process example-level data, which is not suitable for most neural-symbolic tasks. ABLkit provides the ``ABLModel`` to solve this problem. This class serves as a unified wrapper for all base models, which enables the learning part to train, test, and predict on example-level data. | |||
Generally, we can simply instantiate an ``ABLModel`` by: | |||
@@ -14,7 +14,7 @@ We use the MNIST Addition task as a quick start example. In this task, pairs of | |||
Working with Data | |||
----------------- | |||
ABL kit requires data in the format of ``(X, gt_pseudo_label, Y)`` where ``X`` is a list of input examples containing instances, | |||
ABLkit requires data in the format of ``(X, gt_pseudo_label, Y)`` where ``X`` is a list of input examples containing instances, | |||
``gt_pseudo_label`` is the ground-truth label of each example in ``X`` and ``Y`` is the ground-truth reasoning result of each example in ``X``. Note that ``gt_pseudo_label`` is only used to evaluate the machine learning model's performance but not to train it. | |||
In the MNIST Addition task, the data loading looks like | |||
@@ -33,7 +33,7 @@ Read more about `preparing datasets <Datasets.html>`_. | |||
Building the Learning Part | |||
-------------------------- | |||
Learning part is constructed by first defining a base model for machine learning. ABL kit offers considerable flexibility, supporting any base model that conforms to the scikit-learn style (which requires the implementation of ``fit`` and ``predict`` methods), or a PyTorch-based neural network (which has defined the architecture and implemented ``forward`` method). | |||
Learning part is constructed by first defining a base model for machine learning. ABLkit offers considerable flexibility, supporting any base model that conforms to the scikit-learn style (which requires the implementation of ``fit`` and ``predict`` methods), or a PyTorch-based neural network (which has defined the architecture and implemented ``forward`` method). | |||
In this example, we build a simple LeNet5 network as the base model. | |||
.. code:: python | |||
@@ -43,7 +43,7 @@ In this example, we build a simple LeNet5 network as the base model. | |||
cls = LeNet5(num_classes=10) | |||
To facilitate uniform processing, ABL kit provides the ``BasicNN`` class to convert a PyTorch-based neural network into a format compatible with scikit-learn models. To construct a ``BasicNN`` instance, aside from the network itself, we also need to define a loss function, an optimizer, and the computing device. | |||
To facilitate uniform processing, ABLkit provides the ``BasicNN`` class to convert a PyTorch-based neural network into a format compatible with scikit-learn models. To construct a ``BasicNN`` instance, aside from the network itself, we also need to define a loss function, an optimizer, and the computing device. | |||
.. code:: python | |||
@@ -98,7 +98,7 @@ Read more about `building the reasoning part <Reasoning.html>`_. | |||
Building Evaluation Metrics | |||
--------------------------- | |||
ABL kit provides two basic metrics, namely ``SymbolAccuracy`` and ``ReasoningMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. | |||
ABLkit provides two basic metrics, namely ``SymbolAccuracy`` and ``ReasoningMetric``, which are used to evaluate the accuracy of the machine learning model's predictions and the accuracy of the ``logic_forward`` results, respectively. | |||
.. code:: python | |||
@@ -12,7 +12,7 @@ Reasoning part | |||
In this section, we will look at how to build the reasoning part, which | |||
leverages domain knowledge and performs deductive or abductive reasoning. | |||
In ABL kit, building the reasoning part involves two steps: | |||
In ABLkit, building the reasoning part involves two steps: | |||
1. Build a knowledge base by creating a subclass of ``KBBase``, which | |||
specifies how to process pseudo-label of an example to the reasoning result. | |||
@@ -28,7 +28,7 @@ Building a knowledge base | |||
------------------------- | |||
Generally, we can create a subclass derived from ``KBBase`` to build our own | |||
knowledge base. In addition, ABL kit also offers several predefined | |||
knowledge base. In addition, ABLkit also offers several predefined | |||
subclasses of ``KBBase`` (e.g., ``PrologKB`` and ``GroundKB``), | |||
which we can utilize to build our knowledge base more conveniently. | |||
@@ -4,7 +4,7 @@ | |||
# You can set these variables from the command line. | |||
SPHINXOPTS = | |||
SPHINXBUILD = sphinx-build | |||
SPHINXPROJ = ABL kit | |||
SPHINXPROJ = ABLkit | |||
SOURCEDIR = . | |||
BUILDDIR = build | |||
@@ -4,7 +4,7 @@ Installation | |||
Install from PyPI | |||
^^^^^^^^^^^^^^^^^ | |||
The easiest way to install ABL kit is using ``pip``: | |||
The easiest way to install ABLkit is using ``pip``: | |||
.. code:: bash | |||
@@ -1,20 +1,20 @@ | |||
ABL kit | |||
ABLkit | |||
======= | |||
**ABL kit** is an efficient Python toolkit for **Abductive Learning (ABL)**. | |||
**ABLkit** is an efficient Python toolkit for **Abductive Learning (ABL)**. | |||
ABL is a novel paradigm that integrates machine learning and | |||
logical reasoning in a unified framework. It is suitable for tasks | |||
where both data and (logical) domain knowledge are available. | |||
.. image:: _static/img/ABL.png | |||
Key Features of ABL kit: | |||
Key Features of ABLkit: | |||
- **Great Flexibility**: Adaptable to various machine learning modules and logical reasoning components. | |||
- **User-Friendly**: Provide **data**, :blue-bold:`model`, and :green-bold:`KB`, and get started with just a few lines of code. | |||
- **High-Performance**: Optimization for high accuracy and fast training speed. | |||
ABL kit encapsulates advanced ABL techniques, providing users with | |||
ABLkit encapsulates advanced ABL techniques, providing users with | |||
an efficient and convenient toolkit to develop dual-driven ABL systems, | |||
which leverage the power of both data and knowledge. | |||
@@ -26,7 +26,7 @@ Installation | |||
Install from PyPI | |||
^^^^^^^^^^^^^^^^^ | |||
The easiest way to install ABL kit is using ``pip``: | |||
The easiest way to install ABLkit is using ``pip``: | |||
.. code:: bash | |||
@@ -14,7 +14,7 @@ import ablkit # noqa: E402,F401 | |||
# -- Project information ----------------------------------------------------- | |||
project = "ABL kit" | |||
project = "ABLkit" | |||
copyright = "LAMDA, 2024" | |||
# -- General configuration --------------------------------------------------- | |||
@@ -9,7 +9,7 @@ | |||
.. toctree:: | |||
:maxdepth: 1 | |||
:caption: Introduction to ABL kit | |||
:caption: Introduction to ABLkit | |||
Intro/Basics | |||
Intro/Quick-Start | |||
@@ -237,7 +237,7 @@ | |||
"source": [ | |||
"from ablkit.data.structures import ListData\n", | |||
"\n", | |||
"# ListData is a data structure provided by ABL kit that can be used to organize data examples\n", | |||
"# ListData is a data structure provided by ABLkit that can be used to organize data examples\n", | |||
"data_examples = ListData()\n", | |||
"# We use the first 1001st and 3001st data examples in the training set as an illustration\n", | |||
"data_examples.X = [X_1000, X_3000]\n", | |||
@@ -282,7 +282,7 @@ | |||
"source": [ | |||
"from ablkit.data.structures import ListData\n", | |||
"\n", | |||
"# ListData is a data structure provided by ABL kit that can be used to organize data examples\n", | |||
"# ListData is a data structure provided by ABLkit that can be used to organize data examples\n", | |||
"data_examples = ListData()\n", | |||
"# We use the first 100 data examples in the training set as an illustration\n", | |||
"data_examples.X = train_X[:100]\n", | |||
@@ -504,7 +504,7 @@ | |||
"name": "python", | |||
"nbconvert_exporter": "python", | |||
"pygments_lexer": "ipython3", | |||
"version": "3.8.18" | |||
"version": "3.8.13" | |||
}, | |||
"orig_nbformat": 4, | |||
"vscode": { | |||
@@ -97,7 +97,7 @@ | |||
"cell_type": "markdown", | |||
"metadata": {}, | |||
"source": [ | |||
"Next, we transform the tabular data to the format required by ABL kit, which is a tuple of (X, gt_pseudo_label, Y). In this task, we treat the attributes as X and the targets as gt_pseudo_label (ground truth pseudo-labels). Y (reasoning results) are expected to be 0, indicating no rules are violated." | |||
"Next, we transform the tabular data to the format required by ABLkit, which is a tuple of (X, gt_pseudo_label, Y). In this task, we treat the attributes as X and the targets as gt_pseudo_label (ground truth pseudo-labels). Y (reasoning results) are expected to be 0, indicating no rules are violated." | |||
] | |||
}, | |||
{ | |||