Working with Storage¶

The S3 storage can be used with any software supporting the S3 protocol. To configure such program it is required to enter the access key and secret key, the storage server address, and specify the protocol type (HTTP or HTTPS).

For full documentation with examples, use this link. The S3 protocol was originally developed for Amazon's service AWS S3.

Endpoints¶

Currently, the Storage service supports three types of endpoints in each location:

Public access point (HTTP and HTTPS): s3.loc.icdc.io.
Internal access point (HTTP and HTTPS): s3.local.loc.icdc.io.
A DirectConnect access point can be configured upon request to the support service.

The internal access point can be used to access the S3 service only from within the account's networks (e.g., from Compute virtual machines) and does not utilize internet traffic (which may have bandwidth limitations).
When creating a bucket, it becomes accessible immediately from all endpoints (i.e., using the internal access point does not protect against access to the bucket through the public access point).

Note

Currently Storage supports only path-style access, which means access to buckets along the path: s3.loc.icdc.io/<bucket>. The DNS-style access format of the form <bucket>.s3.loc.icdc.io is not currently supported.

S3 Browser (GUI)¶

S3 Browser - a client for working with S3-compatible object storages, which is used to create and delete buckets, view your objects in the storage, and also to upload new objects to S3 storage.
To install S3 Browser, you need to download the distribution kit from developer’s website.

With the first launch, S3 Browser opens a window for configuring access to the storage. In order to do that, input the data from the S3 user page (Overview tab):

Account type: select S3 Compatible Storage.
REST Endpoint: specify the storage address, for example: s3.loc.icdc.io, where loc is the name of the location.
Access Key ID: Access Key ID that you received when creating the cart (bucket).
Secret Access Key: the Secret Key that you received when creating the cart (bucket).
Use secure transfer (SSL/TLS): required option to use an encrypted SSL/TLS channel to interact with the storage.
Advanced Settings > Addressing model: select option Path-style

After pressing Add new account, the configuration of s3browser for working with S3 will be completed.
Instructions for further work with S3 Browser can be found on developer’s website.

s3cmd (CLI)¶

s3cmd is a command line tool and console client for managing S3 object stores.
To install S3cmd, you need to download the distribution from developer's website.

To configure S3cmd, use the s3cmd --configure command and fill in the values for the following parameters:

Access Key: Access Key identifier that you received when creating the bucket.
Secret Key: the Secret Key that you received when creating the bucket.
Default Region: leave blank.
S3 Endpoint: you can specify the public storage address s3.loc.icdc.io or the internal address s3.local.loc.icdc.io, where loc is the name of the location.
DNS-style bucket+hostname:port template for accessing a bucket: specify %(bucket)s.s3.loc.icdc.io (but DNS-style is not supported at the moment).

The s3cmd --configure command will save the settings in the ~/.s3cfg file in the format:

[default]
access_key = ACCESS_KEY
secret_key = SECRET_KEY
host_base = https://s3.loc.icdc.io
host_bucket = %(bucket)s.s3.loc.icdc.io

Public access¶

An example of outputting a file to public access using the s3cmd command:

Example

# 1. Configure keys of S3 user created in Storage into file ~/.s3cfg
# 2. Make Bucket with public ACL
$ s3cmd mb --acl-public s3://test-anon

$ echo "hello world" > index.html

# 3. Save object with public ACL
$ s3cmd put --acl-public index.html s3://test-anon
upload: 'index.html' -> 's3://test-anon/index.html'  [1 of 1]
12 of 12   100% in    0s    18.92 B/s  done
Public URL of the object is: http://s3.loc.icdc.io/test-anon/index.html

$ s3cmd info s3://test-anon/index.html
s3://test-anon/index.html (object):
  File size: 12
  Last mod:  Wed, 25 Oct 2023 08:51:52 GMT
  MIME type: text/plain
  Storage:   STANDARD
  MD5 sum:   6f5902ac237024bdd0c176cb93063dc4
  SSE:       none
  Policy:    none
  CORS:      none
  ACL:       *anon*: READ
  ACL:       test123___icdc___user@example.com: FULL_CONTROL
  URL:       http://s3.loc.icdc.io/test-anon/index.html
  x-amz-meta-s3cmd-attrs: atime:1698223852/ctime:1698223852/gid:1000/gname:danix/md5:6f5902ac237024bdd0c176cb93063dc4/mode:33188/mtime:1698223852/uid:1000/uname:dan

The file is available at the following address: https://s3.loc.icdc.io/test-anon/index.html

S3 bucket policies¶

The ability to restrict access is configured within the S3 protocol using S3 policies.

$ cat > examplepol
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": ["arn:aws:iam::usfolks:user/fred"]},
    "Action": "s3:PutObjectAcl",
    "Resource": [
      "arn:aws:s3:::happybucket/*"
    ]
  }]
}

$ s3cmd setpolicy examplepol s3://happybucket
$ s3cmd delpolicy s3://happybucket