kubeedge
/
sedna
Not watched
1
0
Fork 0
Code
Releases 12 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
131 Commits
4 Branches
121 MB
0 Download
 
 
 
 
 
 
Tree: 95f6432a0c
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '95f6432a0c'
${ noResults }
sedna/scripts/storage-initializer
llhuii 95f6432a0c Makefile: add push for storage-initializer image 4 years ago
..
Dockerfile Fix the issues of storage download script 4 years ago
README.md Fix the issues of storage download script 4 years ago
build_image.sh Makefile: add push for storage-initializer image 4 years ago
download.py Parallel downloading s3 objects in download.py 4 years ago
push_image.sh Makefile: add push for storage-initializer image 4 years ago
requirements.txt Add storage-initializer code/Dockerfile 4 years ago

README.md

storage initializer download script

usage

  1. s3 with public access:
python3 download.py s3://models/classification/model.tar.gz /tmp/models/
# we then download model.tar.gz and extract it into /tmp/models/
  1. s3 with ak/sk:
export S3_ENDPOINT_URL=https://play.min.io
export ACCESS_KEY_ID=Q3AM3UQ867SPQQA43P2F
export SECRET_ACCESS_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
python3 download.py s3://datasets/mnist /tmp/mnist
# we then download the content of mnist directory into /tmp/mnist/
  1. http server:
python3 download.py http://192.168.3.20/model.pb /tmp/models/
# we then download model.pb into /tmp/models/
  1. multi downloads:
python3 download.py http://192.168.3.20/model.pb /tmp/models/ s3://datasets/mnist /tmp/mnist/
# we then download model.pb into /tmp/models/
# and mnist into /tmp/mnist
  1. indirect download(only support s3-compatible):
    the content of s3://datasets/mnist-index.txt:
# this first uncomment line is the directory
s3://datasets
mnist/0.jpeg
mnist/1.jpeg
mnist/2.jpeg

export S3_ENDPOINT_URL=https://play.min.io
export ACCESS_KEY_ID=Q3AM3UQ867SPQQA43P2F
export SECRET_ACCESS_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

# note the char @ indicates it's indirect
python3 download.py s3://datasets/mnist-index.txt @/tmp/mnist
# then only download the mnist/{0,1,2}.jpeg into /tmp/mnist/
Delete task

Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.

cancel
confirm

No Description

Go Python Shell Text Markdown other