s3cmd and Walrus for your private object storage

Posted: 2013/12/26 09:27:05

What is s3cmd?

s3cmd is the command line tool to access files in the Amazon S3 object storage. It is written in Python.

What is Walrus?

Walrus is the object storage of Eucalyptus.

Few more terms

Objects : All files which are stored in the object storage are known as objects.

Key : The filename of an object is known as key.

Bucket : It is the name of the place where people store the files, it behaves like directory with some limitations. Buckets must have unique names.

Install the latest s3cmd from Eucalyptus fork

Install latest version of s3cmd using git in a virtual environment.

$ virtualenv s3
$ source s3/bin/activate
$ git clone https://github.com/eucalyptus/s3cmd
$ cd s3cmd
$ python setup.py install

Configuration file

The default configuration filename is .s3cfg

Following is a minimal example.

[default]
access_key = MRKEEDJHJKQRLMV3K2FB94
secret_key = 3Jy1VKDZmVpwdsffdf8d2PsmfhVojfAW8RmFO2FD
host_base = 192.168.1.100:8773
host_bucket = 192.168.1.100:8773
service_path = /services/Walrus

access_key and secret_key are from your eucarc file. You also know the ip of the host.

To list buckets

$ s3cmd ls
2013-12-24 16:27  s3://foolto
2013-10-19 13:56  s3://mybucket
2013-10-24 15:35  s3://official
2013-12-03 14:47  s3://snapset-5599d8c1-296f-480f-b2be-1e2b03933e42

To list contents of a bucket

$ s3cmd ls s3://foolto
                       DIR   s3://foolto//t2/
2013-12-24 16:51     52680   s3://foolto/lscpu

To create a new bucket

$ s3cmd mb s3://testbucket
Bucket 's3://testbucket/' created

Uploading a file to the bucket

$ s3cmd put .s3cfg s3://testbucket/sample-configuration.txt
.s3cfg -> s3://testbucket/sample-configuration.txt  [1 of 1]
213 of 213   100% in    0s   668.66 B/s  done

In this example we uploaded the configuration file to the testbucket. We can list the bucket content once more.

$ s3cmd ls s3://testbucket/
2013-12-26 04:57       213   s3://testbucket/sample-configuration.txt

Downloading an object from the bucket

$ s3cmd get s3://testbucket/sample-configuration.txt
s3://testbucket/sample-configuration.txt -> ./sample-configuration.txt  [1 of 1]
213 of 213   100% in    0s  1315.45 B/s  done

Delete an object in the bucket

$ s3cmd del s3://testbucket/sample-configuration.txt
File s3://testbucket/sample-configuration.txt deleted

You can learn more about s3cmd from here. If you want to learn about more configuration values available, you can check this link.

Comments

Sessions on Eucalyptus

Posted: 2013/12/24 13:36:13

Yesterday we had a session on Eucalyptus in my house. Total 7 people attended the session including me. We started with an all-in-one cloud installation on the Inter NUC(s). After the cloud is up and running, I installed Fedora 20 cloud image on it.

During the installation we had some nice discussions around different technology choices and features of Eucalyptus. Few people also noted the key differences and similarities between OpenStack and Eucalyptus.

Today evening we had a session on “Open Cloud” using Eucalyptus Community Cloud on #dgplug channel on freenode. 15 people attended the session. We went through the different parts of the user console. People created security groups and key pairs. Everyone started their own instances (with little hiccups) and sshed into them. The UI is simple enough for the students to get the idea.

In future I will be doing more sessions on IRC, starting from installing your own private cloud to learning different technology using cloud. I will also put the notes on my blog. If you think I should cover any particular piece

Comments

Contents © 2004-2014 Kushal Das - Powered by Shonku