s3cmd and Walrus for your private object storage

What is s3cmd?

s3cmd is the command line tool to access files in the Amazon S3 object storage. It is written in Python.

What is Walrus?

Walrus is the object storage of Eucalyptus.

Few more terms

Objects : All files which are stored in the object storage are known as objects.

Key : The filename of an object is known as key.

Bucket : It is the name of the place where people store the files, it behaves like directory with some limitations. Buckets must have unique names.

Install the latest s3cmd from Eucalyptus fork

Install latest version of s3cmd using git in a virtual environment.

$ virtualenv s3
$ source s3/bin/activate
$ git clone https://github.com/eucalyptus/s3cmd
$ cd s3cmd
$ python setup.py install

Configuration file

The default configuration filename is .s3cfg

Following is a minimal example.

[default]
access_key = MRKEEDJHJKQRLMV3K2FB94
secret_key = 3Jy1VKDZmVpwdsffdf8d2PsmfhVojfAW8RmFO2FD
host_base = 192.168.1.100:8773
host_bucket = 192.168.1.100:8773
service_path = /services/Walrus

access_key and secret_key are from your eucarc file. You also know the ip of the host.

To list buckets

$ s3cmd ls
2013-12-24 16:27  s3://foolto
2013-10-19 13:56  s3://mybucket
2013-10-24 15:35  s3://official
2013-12-03 14:47  s3://snapset-5599d8c1-296f-480f-b2be-1e2b03933e42

To list contents of a bucket

$ s3cmd ls s3://foolto
                       DIR   s3://foolto//t2/
2013-12-24 16:51     52680   s3://foolto/lscpu

To create a new bucket

$ s3cmd mb s3://testbucket
Bucket 's3://testbucket/' created

Uploading a file to the bucket

$ s3cmd put .s3cfg s3://testbucket/sample-configuration.txt
.s3cfg -> s3://testbucket/sample-configuration.txt  [1 of 1]
213 of 213   100% in    0s   668.66 B/s  done

In this example we uploaded the configuration file to the testbucket. We can list the bucket content once more.

$ s3cmd ls s3://testbucket/
2013-12-26 04:57       213   s3://testbucket/sample-configuration.txt

Downloading an object from the bucket

$ s3cmd get s3://testbucket/sample-configuration.txt
s3://testbucket/sample-configuration.txt -> ./sample-configuration.txt  [1 of 1]
213 of 213   100% in    0s  1315.45 B/s  done

Delete an object in the bucket

$ s3cmd del s3://testbucket/sample-configuration.txt
File s3://testbucket/sample-configuration.txt deleted

You can learn more about s3cmd from here. If you want to learn about more configuration values available, you can check this link.