Browse Source

[DOC] remove BaseDataElement

pull/1/head
Gao Enhao 1 year ago
parent
commit
66c34aeada
3 changed files with 41 additions and 12 deletions
  1. +37
    -7
      abl/data/structures/list_data.py
  2. +1
    -2
      docs/Intro/Basics.rst
  3. +3
    -3
      docs/Intro/Datasets.rst

+ 37
- 7
abl/data/structures/list_data.py View File

@@ -18,13 +18,43 @@ IndexType = Union[str, slice, int, list, LongTypeTensor, BoolTypeTensor, np.ndar
# https://github.com/open-mmlab/mmdetection/blob/master/mmdet/core/data_structures/instance_data.py # noqa
class ListData(BaseDataElement):
"""
Data structure for example-level data.

Subclass of :class:`BaseDataElement`. All value in `data_fields`
should have the same length. This design refer to
https://github.com/facebookresearch/detectron2/blob/master/detectron2/structures/instances.py

ListData supports `index` and `slice` for data field. The type of value in data field can be either `None` or `list` of base data structures such as `torch.Tensor`, `numpy.ndarray`, `list`, `str` and `tuple`.
Abstract Data Interface used throughout the ABL-Package.

`ListData` is the underlying data structure used in the ABL-Package,
designed to manage diverse forms of data dynamically generated throughout the
Abductive Learning (ABL) framework. This includes handling raw data, predicted
pseudo-labels, abduced pseudo-labels, pseudo-label indices, etc.

As a fundamental data structure in ABL, `ListData` is essential for the smooth
transfer and manipulation of data across various components of the ABL framework,
such as prediction, abductive reasoning, and training phases. It provides a
unified data format across these stages, ensuring compatibility and flexibility
in handling diverse data forms in the ABL framework.

The attributes in ``ListData`` are divided into two parts,
the ``metainfo`` and the ``data`` respectively.

- ``metainfo``: Usually used to store basic information about data examples,
such as symbol number, image size, etc. The attributes can be accessed or
modified by dict-like or object-like operations, such as ``.`` (for data
access and modification), ``in``, ``del``, ``pop(str)``, ``get(str)``,
``metainfo_keys()``, ``metainfo_values()``, ``metainfo_items()``,
``set_metainfo()`` (for set or change key-value pairs in metainfo).

- ``data``: raw data, labels, predictions, and abduced results are stored.
The attributes can be accessed or modified by dict-like or object-like operations,
such as ``.``, ``in``, ``del``, ``pop(str)``, ``get(str)``, ``keys()``,
``values()``, ``items()``. Users can also apply tensor-like
methods to all :obj:`torch.Tensor` in the ``data_fields``, such as ``.cuda()``,
``.cpu()``, ``.numpy()``, ``.to()``, ``to_tensor()``, ``.detach()``.

ListData supports `index` and `slice` for data field. The type of value in
data field can be either `None` or `list` of base data structures such as
`torch.Tensor`, `numpy.ndarray`, `list`, `str` and `tuple`.

This design is inspired by and extends the functionalities of the `BaseDataElement`
class implemented in MMEngine.
https://github.com/open-mmlab/mmengine/blob/main/mmengine/structures/base_data_element.py # noqa E501

Examples:
>>> from abl.data.structures import ListData


+ 1
- 2
docs/Intro/Basics.rst View File

@@ -22,8 +22,7 @@ AI: data, models, and knowledge.
.. image:: ../img/ABL-Package.png

**Data** part manages the storage, operation, and evaluation of data.
It first features class ``ListData`` (derived from base class
``BaseDataElement``), which defines the data structures used in
It first features class ``ListData``, which defines the data structures used in
Abductive Learning, and comprises common data operations like insertion,
deletion, retrieval, slicing, etc. Additionally, a series of Evaluation
Metrics, including class ``SymbolAccuracy`` and ``ReasoningMetric`` (both


+ 3
- 3
docs/Intro/Datasets.rst View File

@@ -53,11 +53,11 @@ As an illustration, in the MNIST Addition example, the data used for training ar
Data Structure
--------------

Besides the user-provided dataset, various forms of data are utilized and dynamicly generate throughout the training and testing process of Abductive Learning framework. Examples include raw data, predicted pseudo-label, abduced pseudo-label, pseudo-label indices, and so on. To manage this diversity and ensure a stable, versatile interface, ABL-Package employs `abstract data interfaces <../API/abl.data.html>`_ to encapsulate different forms of data that will be used in the total learning process.
Besides the user-provided dataset, various forms of data are utilized and dynamicly generated throughout the training and testing process of Abductive Learning framework. Examples include raw data, predicted pseudo-label, abduced pseudo-label, pseudo-label indices, and so on. To manage this diversity and ensure a stable, versatile interface, ABL-Package employs `abstract data interfaces <../API/abl.data.html#data-structure>`_ to encapsulate different forms of data that will be used in the total learning process.

``BaseDataElement`` is the base class for all abstract data interfaces. Inherited from ``BaseDataElement``, ``ListData`` is the most commonly used abstract data interface in ABL-Package. As the fundamental data structure, ``ListData`` implements commonly used data manipulation methods and is responsible for transferring data between various components of ABL, ensuring that stages such as prediction, training, and abductive reasoning can utilize ``ListData`` as a unified input format.
``ListData`` is the underlying abstract data interface utilized in ABL-Package. As the fundamental data structure, ``ListData`` implements commonly used data manipulation methods and is responsible for transferring data between various components of ABL, ensuring that stages such as prediction, abductive reasoning, and training can utilize ``ListData`` as a unified input format.

Before proceeding to other stages, user-provided datasets are firstly converted into ``ListData``. For flexibility, ABL-Package also allows user to directly supply data in ``ListData`` format, which similarly requires the inclusion of three attributes: ``X``, ``gt_pseudo_label``, and ``Y``. The following code shows the basic usage of ``ListData``. More information can be found in the `API documentation <../API/abl.data.html>`_.
Before proceeding to other stages, user-provided datasets are firstly converted into ``ListData``. For flexibility, ABL-Package also allows user to directly supply data in ``ListData`` format, which similarly requires the inclusion of three attributes: ``X``, ``gt_pseudo_label``, and ``Y``. The following code shows the basic usage of ``ListData``. More information can be found in the `API documentation <../API/abl.data.html#data-structure>`_.

.. code-block:: python



Loading…
Cancel
Save