sedna/scripts/storage-initializer/README.md

• usage

storage initializer download script

usage

1. s3 with public access:

python3 download.py s3://models/classification/model.tar.gz /tmp/models/
# we then download model.tar.gz and extract it into /tmp/models/

2. s3 with ak/sk:

export S3_ENDPOINT_URL=https://play.min.io
export ACCESS_KEY_ID=Q3AM3UQ867SPQQA43P2F
export SECRET_ACCESS_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

python3 download.py s3://datasets/mnist/1.jpg /tmp
# we then download the file 1.jpg into /tmp, and result is /tmp/1.jpg.

python3 download.py s3://datasets/mnist /tmp
# we then download the folder mnist into /tmp, and result is /tmp/mnist.

3. http server:

python3 download.py http://192.168.3.20/model.pb /tmp/models/
# we then download model.pb into /tmp/models/

4. multi downloads:

python3 download.py http://192.168.3.20/model.pb /tmp/models/ s3://datasets/mnist /tmp/mnist/
# we then download model.pb into /tmp/models/
# and mnist into /tmp/mnist

5. indirect download(only support s3-compatible):
   the content of s3://datasets/mnist-index.txt:

# this first uncomment line is the directory
s3://datasets
mnist/0.jpeg
mnist/1.jpeg
mnist/2.jpeg

export S3_ENDPOINT_URL=https://play.min.io
export ACCESS_KEY_ID=Q3AM3UQ867SPQQA43P2F
export SECRET_ACCESS_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

# note the char @ indicates it's indirect
python3 download.py s3://datasets/mnist-index.txt @/tmp/mnist
# then only download the mnist/{0,1,2}.jpeg into /tmp/mnist/