Merge pull request #178 from JoeyHwong-gk/docs

Update docs: add lib development guide

build/crd-samples/sedna/lifelonglearningjobv1alpha1.yaml

@@ -1,4 +1,3 @@
-kubectl create -f - <<EOF
 apiVersion: sedna.io/v1alpha1
 kind: LifelongLearningJob
 metadata:

docs/contributing/lib/datasets.md

@@ -0,0 +1,131 @@
+# Dataset Development Guide
+
+## Introduction
+
+The Sedna provides interfaces and public methods related to data conversion and sampling in the Dataset class. The user data processing class can inherit from the Dataset class and use these public capabilities.
+
+
+### 1. Example
+
+The following describes how to use the Dataset by using a `txt-format contain sets of images` as an example. The procedure is as follows:
+
+- 1.1. All dataset classes of Sedna are inherited from the base class `sedna.datasources.BaseDataSource`. The base class BaseDataSource defines the interfaces required by the dataset, provides attributes such as data_parse_func, save, and concat, and provides default implementation. The derived class can reload these default implementations as required.
+
+```python
+    
+    class BaseDataSource:
+        """
+        An abstract class representing a :class:`BaseDataSource`.
+    
+        All datasets that represent a map from keys to data samples should subclass
+        it. All subclasses should overwrite parse`, supporting get train/eval/infer
+        data by a function. Subclasses could also optionally overwrite `__len__`,
+        which is expected to return the size of the dataset.overwrite `x` for the
+        feature-embedding, `y` for the target label.
+    
+        Parameters
+        ----------
+        data_type : str
+            define the datasource is train/eval/test
+        func: function
+            function use to parse an iter object batch by batch
+        """
+    
+        def __init__(self, data_type="train", func=None):
+            self.data_type = data_type  # sample type: train/eval/test
+            self.process_func = None
+            if callable(func):
+                self.process_func = func
+            elif func:
+                self.process_func = ClassFactory.get_cls(
+                    ClassType.CALLBACK, func)()
+            self.x = None  # sample feature
+            self.y = None  # sample label
+            self.meta_attr = None  # special in lifelong learning
+    
+        def num_examples(self) -> int:
+            return len(self.x)
+    
+        def __len__(self):
+            return self.num_examples()
+    
+        def parse(self, *args, **kwargs):
+            raise NotImplementedError
+    
+        @property
+        def is_test_data(self):
+            return self.data_type == "test"
+    
+        def save(self, output=""):
+            return FileOps.dump(self, output)
+
+    class TxtDataParse(BaseDataSource, ABC):
+        """
+        txt file which contain image list parser
+        """
+    
+        def __init__(self, data_type, func=None):
+            super(TxtDataParse, self).__init__(data_type=data_type, func=func)
+    
+        def parse(self, *args, **kwargs):
+            pass
+    
+```
+
+- 1.2. Defining Dataset parse function
+
+```python
+    def parse(self, *args, **kwargs):
+        x_data = []
+        y_data = []
+        use_raw = kwargs.get("use_raw")
+        for f in args:
+            with open(f) as fin:
+                if self.process_func:
+                    res = list(map(self.process_func, [
+                               line.strip() for line in fin.readlines()]))
+                else:
+                    res = [line.strip().split() for line in fin.readlines()]
+            for tup in res:
+                if not len(tup):
+                    continue
+                if use_raw:
+                    x_data.append(tup)
+                else:
+                    x_data.append(tup[0])
+                    if not self.is_test_data:
+                        if len(tup) > 1:
+                            y_data.append(tup[1])
+                        else:
+                            y_data.append(0)
+        self.x = np.array(x_data)
+        self.y = np.array(y_data)
+```
+
+
+### 2. Commissioning
+
+The preceding implementation can be directly used in the PipeStep in Sedna or independently invoked. The code for independently invoking is as follows:
+
+```python
+import os
+import unittest
+
+
+def _load_txt_dataset(dataset_url):
+    # use original dataset url,
+    # see https://github.com/kubeedge/sedna/issues/35
+    return os.path.abspath(dataset_url)
+
+
+class TestDataset(unittest.TestCase):
+
+    def test_txtdata(self):
+        train_data = TxtDataParse(data_type="train", func=_load_txt_dataset)
+        train_data.parse(train_dataset_url, use_raw=True)
+        self.assertEqual(len(train_data), 1)
+
+
+if __name__ == "__main__":
+    unittest.main()
+```

docs/contributing/lib/development.md

@@ -0,0 +1,11 @@
+# Development Guide
+
+This document is intended to provide contributors with an introduction to developing a runnable algorithm module of the Sedna project. 
+
+The Sedna framework components are decoupled and the registration mechanism is used to combine functional components to facilitate function and algorithm expansion. For details about the Sedna architecture and main mechanisms, see [Lib README](/lib/sedna/README.md).
+
+During Sedna application development, the first problem encountered is how to import service data sets to Sedna. For details, see [Datasets Guide](./datasets.md).
+
+For different algorithms, see [Algorithm Development Guide](./new_algorithm.md). You can add new algorithms to Sedna step by step based on the examples provided in this document.
+
+Before develop a module, follow [lib API Reference](https://sedna.readthedocs.io/en/latest/autoapi/lib/sedna/index.html) to learn about the interface design of sedna.

docs/contributing/lib/new_algorithm.md

@@ -0,0 +1,48 @@
+# Algorithm Development Guide
+
+New algorithms, such as `hard example mining` in `incremental_learning` and `joint_inference`, `aggreagtion` in `federated_learning`, `multiple task learning` and `unseen task detect` in `lifelong learning`, need to be extended based on the basic classes provided by Sedna.
+## 1. Add an hard example mining algorithm
+
+The algorithm named `Threshold` is used as an example to describe how to add an HEM algorithm to the Sedna hard example mining algorithm library.
+
+### 1.1 Starting from the `class_factory.py`
+
+First, let's start from the `class_factory.py`. Two classes are defined in `class_factory.py`, namely `ClassType` and `ClassFactory`.
+
+`ClassFactory` can register the modules you want to reuse through decorators. For the new `ClassType.HEM` algorithm, the code is as follows:
+
+```python
+
+@ClassFactory.register(ClassType.HEM, alias="Threshold")
+class ThresholdFilter(BaseFilter, abc.ABC):
+    def __init__(self, threshold=0.5, **kwargs):
+        self.threshold = float(threshold)
+
+    def __call__(self, infer_result=None):
+        # if invalid input, return False
+        if not (infer_result
+                and all(map(lambda x: len(x) > 4, infer_result))):
+            return False
+
+        image_score = 0
+
+        for bbox in infer_result:
+            image_score += bbox[4]
+
+        average_score = image_score / (len(infer_result) or 1)
+        return average_score < self.threshold
+
+```
+
+## 2. Configuring in the CRD yaml
+
+After registration, you only need to change the name of the hem and parameters in the yaml file, and then the corresponding class will be automatically called according to the name.
+
+```yaml
+deploySpec:
+    hardExampleMining:
+      name: "Threshold"
+      parameters:
+        - key: "threshold"
+          value: "0.9"
+```

docs/contributing/prepare-environment.md

@@ -64,7 +64,7 @@ Please follow [the kubeedge instructions][kubeedge] to install KubeEdge.
 Once you've set up the prerequisites, continue with:
 - See [control plane development guide]
 for more details about how to build & test Sedna.
-- See [lib development guide TBD] for more details about how to develop AI algorithms and worker images based on [sedna lib code](/lib).
+- See [lib development guide] for more details about how to develop AI algorithms and worker images based on [sedna lib code](/lib).
 
 [git]: https://git-scm.com/
 [framework]: /docs/proposals/architecture.md#architecture
@@ -76,5 +76,5 @@ for more details about how to build & test Sedna.
 [kind]: https://kind.sigs.k8s.io
 [kubeedge]: https://kubeedge.io/en/docs/
 [kubeedge-k8s-compatibility]: https://github.com/kubeedge/kubeedge#kubernetes-compatibility
-
+[lib development guide]: ./lib/development.md 
 [control plane development guide]: ./control-plane/development.md

docs/index.rst

@@ -58,13 +58,15 @@ Sedna can simply enable edge-cloud synergy capabilities to existing training and
     :titlesonly:
     :glob:
 
-    api/crd/*
     api/lib/*
 
 .. toctree::
+    :maxdepth: 1
     :caption: Contributing
+    :titlesonly:
+    :glob:
 
-    Prepare <contributing/prepare-environment>
+    Control Plane <contributing/prepare-environment>
 
 
 .. toctree::

docs/quickstart.md

@@ -13,32 +13,463 @@ You can find the latest Sedna release [here](https://github.com/kubeedge/sedna/r
 
 ### Deploying Sedna
 
-Please refer to this [link](setup/install.md).
+Sedna provides two deployment methods, which can be selected according to your actual situation:
 
-### Prepare
+- Install Sedna on a cluster Step By Step: [guide here](setup/install.md).
+- Install Sedna AllinOne : [guide here](setup/local-up.md).
 
-todo
+### Component
+Sedna consists of the following components：
 
-### Configure
+![Architecture](./proposals/images/framework.png)
 
-todo
+#### GlobalManager
+* Unified edge-cloud synergy AI task management
+* Cross edge-cloud synergy management and collaboration
+* Central Configuration Management
 
-### Run
+#### LocalController
+* Local process control of edge-cloud synergy AI tasks
+* Local general management: model, dataset, and status synchronization
 
-todo
 
-### Monitor
+#### Worker
+* Do inference or training, based on existing ML framework.
+* Launch on demand, imagine they are docker containers.
+* Different workers for different features.
+* Could run on edge or cloud.
 
-todo
+
+#### Lib
+* Expose the Edge AI features to applications, i.e. training or inference programs.
+
+
+### System Design
+
+There are three stages in a [incremental learning job](./proposals/incremental-learning.md): train/eval/deploy.
+
+![](./proposals/images/incremental-learning-state-machine.png)
+
+## Deployment Guide
+
+### 1. Prepare
+
+#### 1.1 Deployment Planning
+
+In this example, there is only one host with two nodes, which had creating a Kubernetes cluster with `kind`.
+
+| NAME  | ROLES | Ip Address                    | Operating System        | Host Configuration | Storage | Deployment Module                                            |
+| ----- | ------- | ----------------------------- | ----------------------- | ------------------ | ------- | ------------------------------------------------------------ |
+| edge-node  | agent,edge   | 192.168.0.233 | Ubuntu 18.04.5 LTS  | 8C16G              | 500G    | LC，lib, inference worker |
+| sedna-control-plane | control-plane,master    | 172.18.0.2                   | Ubuntu 20.10  | 8C16G              | 500G    | GM，LC，lib，training worker，evaluate worker |
+
+#### 1.2 Network Requirements
+
+In this example the node **sedna-control-plane** has a internal-ip `172.18.0.2`, and **edge-node** can access it.
+
+### 2. Project Deployment
+
+#### 2.1 (optional) create virtual env
+
+```bash
+python3.6 -m venv venv
+source venv/bin/activate
+pip3 install -U pip
+```
+
+#### 2.2 install sedna SDK
+
+```bash
+cd $SENDA_ROOT/lib
+python3.6 setup.py bdist_wheel
+pip3 install dist/sedna*.whl
+``` 
+
+#### 2.3 Prepare your machine learning model and datasets
+
+##### 2.3.1 Encapsulate an Estimators 
+
+Sedna implements several pre-made Estimators in [example](/examples), your can find them from the python scripts called `interface`.
+Sedna supports the Estimators build from popular AI frameworks, such as TensorFlow, Pytorch, PaddlePaddle, MindSpore. Also Custom estimators can be used according to our interface document. 
+All Estimators—pre-made or custom ones—are classes should encapsulate the following actions:
+- Training
+- Evaluation
+- Prediction
+- Export/load
+
+Follow [here](/lib/sedna/README.md) for more details, a [toy_example](/examples/incremental_learning/helmet_detection/training/interface.py) like:
+
+
+```python
+
+os.environ['BACKEND_TYPE'] = 'TENSORFLOW'
+
+class Estimator:
+
+    def __init__(self, **kwargs):
+        ...
+    
+    def train(self, train_data, valid_data=None, **kwargs):     
+        ...
+    
+    def evaluate(self, data, **kwargs):
+       ...
+
+    def predict(self, data, **kwargs):
+        ...
+
+    def load(self, model_url, **kwargs):
+        ...
+
+    def save(self, model_path, **kwargs):
+        ...
+
+    def get_weights(self):
+        ...
+
+    def set_weights(self, weights):
+        ...
+```
+
+##### 2.3.2 Dataset prepare
+
+In incremental_learning jobs, the following files will be indispensable:
+
+- base model: tensorflow object detection Fine-tuning a model from an existing checkpoint.
+- deploy model:  tensorflow object detection model, for inference.
+- train data: images with label use for Fine-tuning model.
+- test data: video stream use for model inference.
+
+```bash
+# download models, including base model and deploy model.
+
+cd /
+wget https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/models.tar.gz
+tar -zxvf models.tar.gz
+
+# download train data
+
+cd /data/helmet_detection  # notes: files here will be monitored and used to trigger the incremental training
+wget  https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/dataset.tar.gz
+tar -zxvf dataset.tar.gz
+
+# download test data
+
+cd /incremental_learning/video/
+wget https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/video.tar.gz
+tar -zxvf video.tar.gz
+ 
+```
+
+#### 2.3.3 Scripts prepare
+
+In incremental_learning jobs, the following scripts will be indispensable:
+
+- train.py: script for model fine-tuning/training.
+- eval.py: script for model evaluate.
+- inference.py: script for data inference.
+
+You can also find demos [here](/examples/incremental_learning/helmet_detection).
+
+Some interfaces should be learn in job pipeline:
+
+- `BaseConfig` provides the capability of obtaining the config from env
+
+```python
+
+from sedna.common.config import BaseConfig
+
+train_dataset_url = BaseConfig.train_dataset_url
+model_url = BaseConfig.model_url
+
+```
+
+- `Context` provides the capability of obtaining the context from CRD
+
+```python
+from sedna.common.config import Context
+
+obj_threshold = Context.get_parameters("obj_threshold")
+nms_threshold = Context.get_parameters("nms_threshold")
+input_shape = Context.get_parameters("input_shape")
+epochs = Context.get_parameters('epochs')
+batch_size = Context.get_parameters('batch_size')
+
+```
+
+- `datasources` base class, as that core feature of sedna require identifying the features and labels from data input, we specify that the first parameter for train/evaluate of the ML framework
+
+```python
+from sedna.datasources import BaseDataSource
+
+
+train_data = BaseDataSource(data_type="train")
+train_data.x = []
+train_data.y = []
+for item in mnist_ds.create_dict_iterator():
+     train_data.x.append(item["image"].asnumpy())
+     train_data.y.append(item["label"].asnumpy())
+```
+
+-  `sedna.core` contain all edge-cloud features,  Please note that each feature has its own parameters.
+- **Hard Example Mining Algorithms** in IncrementalLearning named `hard_example_mining`  
+
+```python
+from sedna.core.incremental_learning import IncrementalLearning
+
+hard_example_mining = IncrementalLearning.get_hem_algorithm_from_config(
+    threshold_img=0.9
+)
+
+# initial an incremental instance
+incremental_instance = IncrementalLearning(
+    estimator=Estimator,
+    hard_example_mining=hem_dict
+)
+
+# Call the interface according to the job state
+
+# train.py
+incremental_instance.train(train_data=train_data, epochs=epochs,
+                          batch_size=batch_size,
+                          class_names=class_names,
+                          input_shape=input_shape,
+                          obj_threshold=obj_threshold,
+                          nms_threshold=nms_threshold)
+
+# inference
+results, _, is_hard_example = incremental_instance.inference(
+            data, input_shape=input_shape)
+            
+
+```
+
+
+### 3. Configuration
+
+##### 3.1 Prepare Image
+This example uses the image:  
+```
+kubeedge/sedna-example-incremental-learning-helmet-detection:v0.4.0
+```
+
+This image is generated by the script [build_images.sh](/examples/build_image.sh), used for creating training, eval and inference worker.  
+
+##### 3.2 Create Incremental Job
+In this example, `$WORKER_NODE` is a custom node, you can fill it which you actually run.  
+
+```
+WORKER_NODE="edge-node" 
+```
+
+- Create Dataset  
+
+```
+kubectl create -f - <<EOF
+apiVersion: sedna.io/v1alpha1
+kind: Dataset
+metadata:
+  name: incremental-dataset
+spec:
+  url: "/data/helmet_detection/train_data/train_data.txt"
+  format: "txt"
+  nodeName: $WORKER_NODE
+EOF
+```
+
+- Create Initial Model to simulate the initial model in incremental learning scenario.  
+
+```
+kubectl create -f - <<EOF
+apiVersion: sedna.io/v1alpha1
+kind: Model
+metadata:
+  name: initial-model
+spec:
+  url : "/models/base_model"
+  format: "ckpt"
+EOF
+```
+
+- Create Deploy Model  
+
+```
+kubectl create -f - <<EOF
+apiVersion: sedna.io/v1alpha1
+kind: Model
+metadata:
+  name: deploy-model
+spec:
+  url : "/models/deploy_model/saved_model.pb"
+  format: "pb"
+EOF
+```
+
+
+### 4. Run
+
+* incremental learning supports hot model updates and cold model updates. Job support 
+cold model updates default. If you want to use hot model updates, please to add the following fields:  
+
+```yaml
+deploySpec:
+  model:
+    hotUpdateEnabled: true
+    pollPeriodSeconds: 60  # default value is 60
+```
+
+* create the job:
+
+```
+IMAGE=kubeedge/sedna-example-incremental-learning-helmet-detection:v0.4.0
+
+kubectl create -f - <<EOF
+apiVersion: sedna.io/v1alpha1
+kind: IncrementalLearningJob
+metadata:
+  name: helmet-detection-demo
+spec:
+  initialModel:
+    name: "initial-model"
+  dataset:
+    name: "incremental-dataset"
+    trainProb: 0.8
+  trainSpec:
+    template:
+      spec:
+        nodeName: $WORKER_NODE
+        containers:
+          - image: $IMAGE
+            name:  train-worker
+            imagePullPolicy: IfNotPresent
+            args: ["train.py"]
+            env:
+              - name: "batch_size"
+                value: "32"
+              - name: "epochs"
+                value: "1"
+              - name: "input_shape"
+                value: "352,640"
+              - name: "class_names"
+                value: "person,helmet,helmet-on,helmet-off"
+              - name: "nms_threshold"
+                value: "0.4"
+              - name: "obj_threshold"
+                value: "0.3"
+    trigger:
+      checkPeriodSeconds: 60
+      timer:
+        start: 02:00
+        end: 20:00
+      condition:
+        operator: ">"
+        threshold: 500
+        metric: num_of_samples
+  evalSpec:
+    template:
+      spec:
+        nodeName: $WORKER_NODE
+        containers:
+          - image: $IMAGE
+            name:  eval-worker
+            imagePullPolicy: IfNotPresent
+            args: ["eval.py"]
+            env:
+              - name: "input_shape"
+                value: "352,640"
+              - name: "class_names"
+                value: "person,helmet,helmet-on,helmet-off"                    
+  deploySpec:
+    model:
+      name: "deploy-model"
+      hotUpdateEnabled: true
+      pollPeriodSeconds: 60 
+    trigger:
+      condition:
+        operator: ">"
+        threshold: 0.1
+        metric: precision_delta
+    hardExampleMining:
+      name: "IBT"
+      parameters:
+        - key: "threshold_img"
+          value: "0.9"
+        - key: "threshold_box"
+          value: "0.9"
+    template:
+      spec:
+        nodeName: $WORKER_NODE
+        containers:
+        - image: $IMAGE
+          name:  infer-worker
+          imagePullPolicy: IfNotPresent
+          args: ["inference.py"]
+          env:
+            - name: "input_shape"
+              value: "352,640"
+            - name: "video_url"
+              value: "file://video/video.mp4"
+            - name: "HE_SAVED_URL" 
+              value: "/he_saved_url"
+          volumeMounts:
+          - name: localvideo
+            mountPath: /video/
+          - name: hedir
+            mountPath: /he_saved_url
+          resources:  # user defined resources
+            limits:
+              memory: 2Gi
+        volumes:   # user defined volumes
+          - name: localvideo
+            hostPath:
+              path: /incremental_learning/video/
+              type: DirectoryorCreate
+          - name: hedir
+            hostPath:
+              path:  /incremental_learning/he/
+              type: DirectoryorCreate
+  outputDir: "/output"
+EOF
+```
+
+1. The `Dataset` describes data with labels and `HE_SAVED_URL` indicates the address of the deploy container for uploading hard examples. Users will mark label for the hard examples in the address.
+2. Ensure that the path of outputDir in the YAML file exists on your node. This path will be directly mounted to the container.
+
+
+### 5. Monitor
+
+### Check Incremental Learning Job
+Query the service status:  
+
+```
+kubectl get incrementallearningjob helmet-detection-demo
+```
+
+In the `IncrementalLearningJob` resource helmet-detection-demo, the following trigger is configured:  
+
+```
+trigger:
+  checkPeriodSeconds: 60
+  timer:
+    start: 02:00
+    end: 20:00
+  condition:
+    operator: ">"
+    threshold: 500
+    metric: num_of_samples
+```
 
 ## API
 
-- CRD: Please refer to this [link](api/crd).
+- control-plane: Please refer to this [link](api/crd).
 - Lib: Please refer to this [link](api/lib).
 
 ## Contributing
 
-Contributions are very welcome! You can see our [CONTRIBUTING](CONTRIBUTING.md) for more information.
+Contributions are very welcome!
+
+- control-plane: Please refer to this [link](contributing/control-plane/development.md).
+- Lib: Please refer to this [link](contributing/lib/development.md).
 
 ## Community
 

docs/related_link.md

@@ -1 +1,9 @@
-[支持边云协同终身学习特性，KubeEdge 子项目 Sedna 0.3.0 版本发布！](https://juejin.cn/post/6970866022286884878)
+[支持边云协同终身学习特性，KubeEdge 子项目 Sedna 0.3.0 版本发布！](https://juejin.cn/post/6970866022286884878)
+
+[【HDC.Cloud 2021】边云协同，打通AI最后一公里](https://xie.infoq.cn/article/b22e72afe8de50ca34269bb21)
+
+[KubeEdge Sedna如何实现边缘AI模型精度提升50%](https://www.huaweicloud.com/zhishi/hdc2021-Track-24-18.html)
+
+[KubeEdge子项目Sedna 0.1版本发布！支持边云协同增量学习、联邦学习、协同推理](https://mp.weixin.qq.com/s/3Ei8ynSAxnfuoIWYdb7Gpg)
+
+[加速AI边云协同创新！KubeEdge社区建立Sedna子项目](https://cloud.tencent.com/developer/article/1791739)