|
|
|
@@ -1,138 +1,75 @@ |
|
|
|
|
|
|
|
# Quick Start |
|
|
|
|
|
|
|
## Introduction |
|
|
|
## Guide |
|
|
|
- If you are new to Sedna, you can try the command step by step in this page. |
|
|
|
- If you have played the following example, you can find more [examples](/examples/README.md). |
|
|
|
- If you want to know more about sedna's architecture and component, you can find them in [sedna home]. |
|
|
|
- If you're looking to contribute documentation improvements, you'll specifically want to see the [kubernetes documentation style guide] before [filing an issue][file-an-issue]. |
|
|
|
- If you're planning to contribute code changes, you'll want to read the [development preparation guide] next. |
|
|
|
- If you're planning to add a new synergy feature directly, you'll want to read the [guide][add-feature-guide] next. |
|
|
|
|
|
|
|
Sedna provide some examples of running Sedna jobs in [here](/examples/README.md) |
|
|
|
When done, you can also refer our [recommended Git workflow] and [pull request best practices] before submitting a pull request. |
|
|
|
|
|
|
|
Here is a general guide to quick start an incremental learning job. |
|
|
|
|
|
|
|
### Get Sedna |
|
|
|
[proposals]: /docs/proposals |
|
|
|
[development preparation guide]: ./prepare-environment.md |
|
|
|
[add-feature-guide]: control-plane/add-a-new-synergy-feature.md |
|
|
|
|
|
|
|
You can find the latest Sedna release [here](https://github.com/kubeedge/sedna/releases). |
|
|
|
[sedna home]: https://github.com/kubeedge/sedna |
|
|
|
[issues]: https://github.com/kubeedge/sedna/issues |
|
|
|
[file-an-issue]: https://github.com/kubeedge/sedna/issues/new/choose |
|
|
|
[file-a-fr]: https://github.com/kubeedge/sedna/issues/new?labels=kind%2Ffeature&template=enhancement.md |
|
|
|
|
|
|
|
### Deploying Sedna |
|
|
|
[github]: https://github.com/ |
|
|
|
[kubernetes documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md |
|
|
|
[recommended Git workflow]: https://github.com/kubernetes/community/blob/master/contributors/guide/github-workflow.md#workflow |
|
|
|
[pull request best practices]: https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews |
|
|
|
[Kubernetes help wanted]: https://www.kubernetes.dev/docs/guide/help-wanted/ |
|
|
|
|
|
|
|
Sedna provides two deployment methods, which can be selected according to your actual situation: |
|
|
|
|
|
|
|
- Install Sedna on a cluster Step By Step: [guide here](setup/install.md). |
|
|
|
- Install Sedna AllinOne : [guide here](setup/local-up.md). |
|
|
|
The following is showing how to run an incremental learning job by sedna. |
|
|
|
## Quick Start |
|
|
|
|
|
|
|
### Component |
|
|
|
Sedna consists of the following components： |
|
|
|
#### 0. Check the Environment |
|
|
|
|
|
|
|
 |
|
|
|
|
|
|
|
#### GlobalManager |
|
|
|
* Unified edge-cloud synergy AI task management |
|
|
|
* Cross edge-cloud synergy management and collaboration |
|
|
|
* Central Configuration Management |
|
|
|
|
|
|
|
#### LocalController |
|
|
|
* Local process control of edge-cloud synergy AI tasks |
|
|
|
* Local general management: model, dataset, and status synchronization |
|
|
|
|
|
|
|
|
|
|
|
#### Worker |
|
|
|
* Do inference or training, based on existing ML framework. |
|
|
|
* Launch on demand, imagine they are docker containers. |
|
|
|
* Different workers for different features. |
|
|
|
* Could run on edge or cloud. |
|
|
|
|
|
|
|
|
|
|
|
#### Lib |
|
|
|
* Expose the Edge AI features to applications, i.e. training or inference programs. |
|
|
|
|
|
|
|
|
|
|
|
### System Design |
|
|
|
|
|
|
|
There are three stages in a [incremental learning job](./proposals/incremental-learning.md): train/eval/deploy. |
|
|
|
|
|
|
|
 |
|
|
|
|
|
|
|
## Deployment Guide |
|
|
|
|
|
|
|
### 1. Prepare |
|
|
|
|
|
|
|
#### 1.1 Deployment Planning |
|
|
|
|
|
|
|
In this example, there is only one host with two nodes, which had creating a Kubernetes cluster with `kind`. |
|
|
|
|
|
|
|
| NAME | ROLES | Ip Address | Operating System | Host Configuration | Storage | Deployment Module | |
|
|
|
| ----- | ------- | ----------------------------- | ----------------------- | ------------------ | ------- | ------------------------------------------------------------ | |
|
|
|
| edge-node | agent,edge | 192.168.0.233 | Ubuntu 18.04.5 LTS | 8C16G | 500G | LC，lib, inference worker | |
|
|
|
| sedna-control-plane | control-plane,master | 172.18.0.2 | Ubuntu 20.10 | 8C16G | 500G | GM，LC，lib，training worker，evaluate worker | |
|
|
|
|
|
|
|
#### 1.2 Network Requirements |
|
|
|
|
|
|
|
In this example the node **sedna-control-plane** has a internal-ip `172.18.0.2`, and **edge-node** can access it. |
|
|
|
|
|
|
|
### 2. Project Deployment |
|
|
|
|
|
|
|
#### 2.1 (optional) create virtual env |
|
|
|
For Sedna all-in-one installation, it requires you: |
|
|
|
- 1 VM **(one machine is OK, cluster is not required)** |
|
|
|
- 2 CPUs or more |
|
|
|
- 2GB+ free memory, depends on node number setting |
|
|
|
- 10GB+ free disk space |
|
|
|
- Internet connection(docker hub, github etc.) |
|
|
|
- Linux platform, such as ubuntu/centos |
|
|
|
- Docker 17.06+ |
|
|
|
|
|
|
|
you can check the docker version by the following command, |
|
|
|
```bash |
|
|
|
python3.6 -m venv venv |
|
|
|
source venv/bin/activate |
|
|
|
pip3 install -U pip |
|
|
|
docker -v |
|
|
|
``` |
|
|
|
after doing that, the output will be like this, that means your version fits the bill. |
|
|
|
``` |
|
|
|
Docker version 19.03.6, build 369ce74a3c |
|
|
|
``` |
|
|
|
|
|
|
|
#### 2.2 install sedna SDK |
|
|
|
|
|
|
|
```bash |
|
|
|
cd $SENDA_ROOT/lib |
|
|
|
python3.6 setup.py bdist_wheel |
|
|
|
pip3 install dist/sedna*.whl |
|
|
|
``` |
|
|
|
|
|
|
|
#### 2.3 Prepare your machine learning model and datasets |
|
|
|
|
|
|
|
##### 2.3.1 Encapsulate an Estimators |
|
|
|
|
|
|
|
Sedna implements several pre-made Estimators in [example](/examples), your can find them from the python scripts called `interface`. |
|
|
|
Sedna supports the Estimators build from popular AI frameworks, such as TensorFlow, Pytorch, PaddlePaddle, MindSpore. Also Custom estimators can be used according to our interface document. |
|
|
|
All Estimators—pre-made or custom ones—are classes should encapsulate the following actions: |
|
|
|
- Training |
|
|
|
- Evaluation |
|
|
|
- Prediction |
|
|
|
- Export/load |
|
|
|
|
|
|
|
Follow [here](/lib/sedna/README.md) for more details, a [toy_example](/examples/incremental_learning/helmet_detection/training/interface.py) like: |
|
|
|
|
|
|
|
|
|
|
|
```python |
|
|
|
|
|
|
|
os.environ['BACKEND_TYPE'] = 'TENSORFLOW' |
|
|
|
|
|
|
|
class Estimator: |
|
|
|
|
|
|
|
def __init__(self, **kwargs): |
|
|
|
... |
|
|
|
|
|
|
|
def train(self, train_data, valid_data=None, **kwargs): |
|
|
|
... |
|
|
|
|
|
|
|
def evaluate(self, data, **kwargs): |
|
|
|
... |
|
|
|
|
|
|
|
def predict(self, data, **kwargs): |
|
|
|
... |
|
|
|
|
|
|
|
def load(self, model_url, **kwargs): |
|
|
|
... |
|
|
|
#### 1. Deploy Sedna |
|
|
|
Sedna provides three deployment methods, which can be selected according to your actual situation: |
|
|
|
|
|
|
|
def save(self, model_path, **kwargs): |
|
|
|
... |
|
|
|
- [Install Sedna AllinOne](setup/all-in-one.md). (used for development, here we use it) |
|
|
|
- [Install Sedna local up](setup/local-up.md). |
|
|
|
- [Install Sedna on a cluster](setup/install.md). |
|
|
|
|
|
|
|
def get_weights(self): |
|
|
|
... |
|
|
|
The [all-in-one script](/scripts/installation/all-in-one.sh) is used to install Sedna along with a mini Kubernetes environment locally, including: |
|
|
|
- A Kubernetes v1.21 cluster with multi worker nodes, default zero worker node. |
|
|
|
- KubeEdge with multi edge nodes, default is latest KubeEdge and one edge node. |
|
|
|
- Sedna, default is the latest version. |
|
|
|
|
|
|
|
def set_weights(self, weights): |
|
|
|
... |
|
|
|
``` |
|
|
|
|
|
|
|
##### 2.3.2 Dataset prepare |
|
|
|
```bash |
|
|
|
curl https://raw.githubusercontent.com/kubeedge/sedna/master/scripts/installation/all-in-one.sh | NUM_EDGE_NODES=2 bash - |
|
|
|
``` |
|
|
|
|
|
|
|
#### 2. Download model and datasets |
|
|
|
In incremental_learning jobs, the following files will be indispensable: |
|
|
|
|
|
|
|
- base model: tensorflow object detection Fine-tuning a model from an existing checkpoint. |
|
|
|
@@ -161,101 +98,8 @@ tar -zxvf video.tar.gz |
|
|
|
|
|
|
|
``` |
|
|
|
|
|
|
|
#### 2.3.3 Scripts prepare |
|
|
|
|
|
|
|
In incremental_learning jobs, the following scripts will be indispensable: |
|
|
|
|
|
|
|
- train.py: script for model fine-tuning/training. |
|
|
|
- eval.py: script for model evaluate. |
|
|
|
- inference.py: script for data inference. |
|
|
|
|
|
|
|
You can also find demos [here](/examples/incremental_learning/helmet_detection). |
|
|
|
|
|
|
|
Some interfaces should be learn in job pipeline: |
|
|
|
|
|
|
|
- `BaseConfig` provides the capability of obtaining the config from env |
|
|
|
|
|
|
|
```python |
|
|
|
#### 3. Create model and dataset object |
|
|
|
|
|
|
|
from sedna.common.config import BaseConfig |
|
|
|
|
|
|
|
train_dataset_url = BaseConfig.train_dataset_url |
|
|
|
model_url = BaseConfig.model_url |
|
|
|
|
|
|
|
``` |
|
|
|
|
|
|
|
- `Context` provides the capability of obtaining the context from CRD |
|
|
|
|
|
|
|
```python |
|
|
|
from sedna.common.config import Context |
|
|
|
|
|
|
|
obj_threshold = Context.get_parameters("obj_threshold") |
|
|
|
nms_threshold = Context.get_parameters("nms_threshold") |
|
|
|
input_shape = Context.get_parameters("input_shape") |
|
|
|
epochs = Context.get_parameters('epochs') |
|
|
|
batch_size = Context.get_parameters('batch_size') |
|
|
|
|
|
|
|
``` |
|
|
|
|
|
|
|
- `datasources` base class, as that core feature of sedna require identifying the features and labels from data input, we specify that the first parameter for train/evaluate of the ML framework |
|
|
|
|
|
|
|
```python |
|
|
|
from sedna.datasources import BaseDataSource |
|
|
|
|
|
|
|
|
|
|
|
train_data = BaseDataSource(data_type="train") |
|
|
|
train_data.x = [] |
|
|
|
train_data.y = [] |
|
|
|
for item in mnist_ds.create_dict_iterator(): |
|
|
|
train_data.x.append(item["image"].asnumpy()) |
|
|
|
train_data.y.append(item["label"].asnumpy()) |
|
|
|
``` |
|
|
|
|
|
|
|
- `sedna.core` contain all edge-cloud features, Please note that each feature has its own parameters. |
|
|
|
- **Hard Example Mining Algorithms** in IncrementalLearning named `hard_example_mining` |
|
|
|
|
|
|
|
```python |
|
|
|
from sedna.core.incremental_learning import IncrementalLearning |
|
|
|
|
|
|
|
hard_example_mining = IncrementalLearning.get_hem_algorithm_from_config( |
|
|
|
threshold_img=0.9 |
|
|
|
) |
|
|
|
|
|
|
|
# initial an incremental instance |
|
|
|
incremental_instance = IncrementalLearning( |
|
|
|
estimator=Estimator, |
|
|
|
hard_example_mining=hem_dict |
|
|
|
) |
|
|
|
|
|
|
|
# Call the interface according to the job state |
|
|
|
|
|
|
|
# train.py |
|
|
|
incremental_instance.train(train_data=train_data, epochs=epochs, |
|
|
|
batch_size=batch_size, |
|
|
|
class_names=class_names, |
|
|
|
input_shape=input_shape, |
|
|
|
obj_threshold=obj_threshold, |
|
|
|
nms_threshold=nms_threshold) |
|
|
|
|
|
|
|
# inference |
|
|
|
results, _, is_hard_example = incremental_instance.inference( |
|
|
|
data, input_shape=input_shape) |
|
|
|
|
|
|
|
|
|
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
### 3. Configuration |
|
|
|
|
|
|
|
##### 3.1 Prepare Image |
|
|
|
This example uses the image: |
|
|
|
``` |
|
|
|
kubeedge/sedna-example-incremental-learning-helmet-detection:v0.4.0 |
|
|
|
``` |
|
|
|
|
|
|
|
This image is generated by the script [build_images.sh](/examples/build_image.sh), used for creating training, eval and inference worker. |
|
|
|
|
|
|
|
##### 3.2 Create Incremental Job |
|
|
|
In this example, `$WORKER_NODE` is a custom node, you can fill it which you actually run. |
|
|
|
|
|
|
|
``` |
|
|
|
@@ -305,140 +149,20 @@ spec: |
|
|
|
EOF |
|
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
### 4. Run |
|
|
|
|
|
|
|
* incremental learning supports hot model updates and cold model updates. Job support |
|
|
|
cold model updates default. If you want to use hot model updates, please to add the following fields: |
|
|
|
|
|
|
|
```yaml |
|
|
|
deploySpec: |
|
|
|
model: |
|
|
|
hotUpdateEnabled: true |
|
|
|
pollPeriodSeconds: 60 # default value is 60 |
|
|
|
``` |
|
|
|
#### 4. Start an incremental learning job |
|
|
|
|
|
|
|
* create the job: |
|
|
|
|
|
|
|
``` |
|
|
|
IMAGE=kubeedge/sedna-example-incremental-learning-helmet-detection:v0.4.0 |
|
|
|
|
|
|
|
kubectl create -f - <<EOF |
|
|
|
apiVersion: sedna.io/v1alpha1 |
|
|
|
kind: IncrementalLearningJob |
|
|
|
metadata: |
|
|
|
name: helmet-detection-demo |
|
|
|
spec: |
|
|
|
initialModel: |
|
|
|
name: "initial-model" |
|
|
|
dataset: |
|
|
|
name: "incremental-dataset" |
|
|
|
trainProb: 0.8 |
|
|
|
trainSpec: |
|
|
|
template: |
|
|
|
spec: |
|
|
|
nodeName: $WORKER_NODE |
|
|
|
containers: |
|
|
|
- image: $IMAGE |
|
|
|
name: train-worker |
|
|
|
imagePullPolicy: IfNotPresent |
|
|
|
args: ["train.py"] |
|
|
|
env: |
|
|
|
- name: "batch_size" |
|
|
|
value: "32" |
|
|
|
- name: "epochs" |
|
|
|
value: "1" |
|
|
|
- name: "input_shape" |
|
|
|
value: "352,640" |
|
|
|
- name: "class_names" |
|
|
|
value: "person,helmet,helmet-on,helmet-off" |
|
|
|
- name: "nms_threshold" |
|
|
|
value: "0.4" |
|
|
|
- name: "obj_threshold" |
|
|
|
value: "0.3" |
|
|
|
trigger: |
|
|
|
checkPeriodSeconds: 60 |
|
|
|
timer: |
|
|
|
start: 02:00 |
|
|
|
end: 20:00 |
|
|
|
condition: |
|
|
|
operator: ">" |
|
|
|
threshold: 500 |
|
|
|
metric: num_of_samples |
|
|
|
evalSpec: |
|
|
|
template: |
|
|
|
spec: |
|
|
|
nodeName: $WORKER_NODE |
|
|
|
containers: |
|
|
|
- image: $IMAGE |
|
|
|
name: eval-worker |
|
|
|
imagePullPolicy: IfNotPresent |
|
|
|
args: ["eval.py"] |
|
|
|
env: |
|
|
|
- name: "input_shape" |
|
|
|
value: "352,640" |
|
|
|
- name: "class_names" |
|
|
|
value: "person,helmet,helmet-on,helmet-off" |
|
|
|
deploySpec: |
|
|
|
model: |
|
|
|
name: "deploy-model" |
|
|
|
hotUpdateEnabled: true |
|
|
|
pollPeriodSeconds: 60 |
|
|
|
trigger: |
|
|
|
condition: |
|
|
|
operator: ">" |
|
|
|
threshold: 0.1 |
|
|
|
metric: precision_delta |
|
|
|
hardExampleMining: |
|
|
|
name: "IBT" |
|
|
|
parameters: |
|
|
|
- key: "threshold_img" |
|
|
|
value: "0.9" |
|
|
|
- key: "threshold_box" |
|
|
|
value: "0.9" |
|
|
|
template: |
|
|
|
spec: |
|
|
|
nodeName: $WORKER_NODE |
|
|
|
containers: |
|
|
|
- image: $IMAGE |
|
|
|
name: infer-worker |
|
|
|
imagePullPolicy: IfNotPresent |
|
|
|
args: ["inference.py"] |
|
|
|
env: |
|
|
|
- name: "input_shape" |
|
|
|
value: "352,640" |
|
|
|
- name: "video_url" |
|
|
|
value: "file://video/video.mp4" |
|
|
|
- name: "HE_SAVED_URL" |
|
|
|
value: "/he_saved_url" |
|
|
|
volumeMounts: |
|
|
|
- name: localvideo |
|
|
|
mountPath: /video/ |
|
|
|
- name: hedir |
|
|
|
mountPath: /he_saved_url |
|
|
|
resources: # user defined resources |
|
|
|
limits: |
|
|
|
memory: 2Gi |
|
|
|
volumes: # user defined volumes |
|
|
|
- name: localvideo |
|
|
|
hostPath: |
|
|
|
path: /incremental_learning/video/ |
|
|
|
type: DirectoryorCreate |
|
|
|
- name: hedir |
|
|
|
hostPath: |
|
|
|
path: /incremental_learning/he/ |
|
|
|
type: DirectoryorCreate |
|
|
|
outputDir: "/output" |
|
|
|
EOF |
|
|
|
kubectl -f https://raw.githubusercontent.com/kubeedge/sedna/main/examples/incremental_learning/helmet_detection/helmet_detection.yaml |
|
|
|
``` |
|
|
|
|
|
|
|
1. The `Dataset` describes data with labels and `HE_SAVED_URL` indicates the address of the deploy container for uploading hard examples. Users will mark label for the hard examples in the address. |
|
|
|
2. Ensure that the path of outputDir in the YAML file exists on your node. This path will be directly mounted to the container. |
|
|
|
|
|
|
|
|
|
|
|
### 5. Monitor |
|
|
|
#### 5. Check the result |
|
|
|
|
|
|
|
### Check Incremental Learning Job |
|
|
|
Query the service status: |
|
|
|
|
|
|
|
``` |
|
|
|
|
Jie Pu
4 years ago