sedna/examples/storage/s3/incremental_learning/README.md

# Using Incremental Learning Job in Helmet Detection Scenario on S3

This example is based on the example: [Using Incremental Learning Job in Helmet Detection Scenario](/examples/incremental_learning/helmet_detection/README.md).    

### Prepare Nodes
Assume you have created a [KubeEdge](https://github.com/kubeedge/kubeedge) cluster that have two cloud nodes(e.g., `cloud-node1`, `cloud-node2`)
and one edge node(e.g., `edge-node`).  

### Create a secret with your S3 user credential.   

```shell
kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    s3-endpoint: s3.amazonaws.com # replace with your s3 endpoint e.g minio-service.kubeflow:9000 
    s3-usehttps: "1" # by default 1, if testing with minio you can set to 0
stringData: # use `stringData` for raw credential string or `data` for base64 encoded string
  ACCESS_KEY_ID: XXXX
  SECRET_ACCESS_KEY: XXXXXXXX
EOF
```

### Prepare Model
* Download [models](https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/model.tar.gz).  
* Put the unzipped model file into the bucket of your cloud storage service.  
* Attach the created secret to the Model and create Model.  

```shell
kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Model
metadata:
  name: initial-model
  spec:
    url : "s3://kubeedge/model/base_model"
    format: "ckpt"
    credentialName: mysecret
EOF
```

```shell
kubectl $action -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Model
metadata:
  name: deploy-model
spec:
  url: "s3://kubeedge/model/deploy_model/saved_model.pb"
  format: "pb"
  credentialName: mysecret
EOF
```

### Prepare Dataset
* Download [dataset](https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/dataset.tar.gz).  
* Put the unzipped dataset file into the bucket of your cloud storage service.  
* Attach the created secret to Dataset and create Dataset.  

```shell
kubectl $action -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: Dataset
metadata:
  name: incremental-dataset
spec:
  url: "s3://kubeedge/data/helmet_detection/train_data/train_data.txt"
  format: "txt"
  nodeName: cloud-node1
  credentialName: mysecret
EOF
```

### Prepare Image
This example uses the image:  

```shell
kubeedge/sedna-example-incremental-learning-helmet-detection:v0.3.1
```

This image is generated by the script [build_images.sh](/examples/build_image.sh), used for creating training, eval and inference worker.  

### Prepare Job
* Inference/Train/Eval worker now can be deployed by nodeName and [nodeSelector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) on multiple nodes.  

* Make sure to follow the local dir which exists in edge side.    

```shell
mkdir -p /incremental_learning/video/
```

* Download [video](https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/video.tar.gz), unzip video.tar.gz, and put it into `/incremental_learning/video/`.    

```
cd /incremental_learning/video/
wget https://kubeedge.obs.cn-north-1.myhuaweicloud.com/examples/helmet-detection/video.tar.gz
tar -zxvf video.tar.gz
```

* Attach the created secret to the Job and create Job.  

```shell
IMAGE=kubeedge/sedna-example-incremental-learning-helmet-detection:v0.3.1  

kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: IncrementalLearningJob
metadata:
  name: helmet-detection-demo
spec:
  initialModel:
    name: "initial-model"
  dataset:
    name: "incremental-dataset"
    trainProb: 0.8
  trainSpec:
    template:
      spec:
        nodeName: cloud-node1
        containers:
          - image: $IMAGE
            name:  train-worker
            imagePullPolicy: IfNotPresent
            args: ["train.py"]
            env:
              - name: "batch_size"
                value: "32"
              - name: "epochs"
                value: "1"
              - name: "input_shape"
                value: "352,640"
              - name: "class_names"
                value: "person,helmet,helmet-on,helmet-off"
              - name: "nms_threshold"
                value: "0.4"
              - name: "obj_threshold"
                value: "0.3"
    trigger:
      checkPeriodSeconds: 60
      timer:
        start: 02:00
        end: 20:00
      condition:
        operator: ">"
        threshold: 500
        metric: num_of_samples
  evalSpec:
    template:
      spec:
        nodeName: cloud-node2
        containers:
          - image: $IMAGE
            name:  eval-worker
            imagePullPolicy: IfNotPresent
            args: ["eval.py"]
            env:
              - name: "input_shape"
                value: "352,640"
              - name: "class_names"
                value: "person,helmet,helmet-on,helmet-off"
  deploySpec:
    model:
      name: "deploy-model"
    trigger:
      condition:
        operator: ">"
        threshold: 0.1
        metric: precision_delta
    hardExampleMining:
      name: "IBT"
      parameters:
        - key: "threshold_img"
          value: "0.9"
        - key: "threshold_box"
          value: "0.9"
    template:
      spec:
        nodeName: edge-node
        containers:
          - image: $IMAGE
            name:  infer-worker
            imagePullPolicy: IfNotPresent
            args: ["inference.py"]
            env:
              - name: "input_shape"
                value: "352,640"
              - name: "video_url"
                value: "file://video/video.mp4"
              - name: "HE_SAVED_URL"
                value: "/he_saved_url"
            volumeMounts:
              - name: localvideo
                mountPath: /video/
              - name: hedir
                mountPath: /he_saved_url
            resources:  # user defined resources
              limits:
                memory: 2Gi
        volumes:   # user defined volumes
          - name: localvideo
            hostPath:
              path: /incremental_learning/video/
              type: DirectoryOrCreate
          - name: hedir
            hostPath:
              path:  /incremental_learning/he/
              type: DirectoryOrCreate
  credentialName: mysecret
  outputDir: "s3://kubeedge/incremental_learning/output"
EOF
```