CORS support

For security reasons Javascript running in a browser is not allowed to make requests to domains other than the one from where it came from. This is referred to as the Same Origin Policy. CORS is specification allowing browsers and application servers work out an agreement whereby these types of requests are allowed.

Swift 1.7.5 introduced CORS support. This means a Javascript application running in a browser and hosted outside of a Swift cluster can still query that cluster’s API. I expect to see lots of Swift based applications over the coming months thanks to this great new feature.

The ETag

Every object written to a Swift cluster has an ETag associated with it. The value of this ETag is the MD5 digest of the file’s contents. What makes this useful is that you can use it to make a conditional request for the object using the If-Match, If-None-Match headers.

For example, lets say your Swift cluster contains the object “movie1.mp4”. You already have a version of the file locally but you’re not sure if it’s exactly the same. You don’t want to have to download it unnecessarily because it would take too long and/or might incur bandwidth charges. What you can do is invoke a conditional download request, i.e.

$ md5sum movie1.mp4
d41d8cd98f00b204e9800998ecf8427e  movie1.mp4

$ curl -i -H 'X-Auth-Token: xxx' -H 'If-None-Match: d41d8cd98f00b204e9800998ecf8427e' http://swift.17od.com/movies/movie1.mp4
HTTP/1.1 304 Not Modified
...

The 304 response code tells us that our local version is the same as the remote version. If it wasn’t the same it would be downloaded.

Object Versioning

With object versioning, each PUT request to an object will result in the existing object being archived to a special “versions” container.

Versioning is controlled at the container level by setting the header “X-Versions-Location” to the name of the container where you want to archive object versions.

For more information see Object Versioning in the developer documentation.

Renaming or Moving an Object

Renaming or moving objects isn’t supported in the classic sense. However the same result can be achieved by downloading the existing object and re-uploading it to the new container and/or with a new name. Of course if the object is large this can be a time consuming operation. Luckily Swift supports server side copies. What this means is this object is copied from it’s source to destination all within the confines of the Swift cluster.

Either PUT or COPY can be used to perform a server side copy. Neither has an advantage over the other.

For example, given the container and object “photos/sunset.jpg”, here’s how to move it to “holiday-pics/sunset_glow.jpg”.

$ curl -X PUT -H 'X-Auth-Token: xxx' -H 'X-Copy-From: /photos/sunset.jpg' http://swift.17od.com/holiday-pics/sunset_glow.jpg

See Copy Object for more details.

Expiring Objects

Objects can be given an expiry time. When that time is reached Swift will automatically delete the object. Object’s are given an expiry time by setting either the X-Delete-At or X-Delete-After headers.

The value given to X-Delete-At is a Unix Epoch timestamp. There are many ways of converting a time to an epoch integer. For example on UNIX you can to,

$ date +%s
1355945201

The epoch converter website allows you to convert any time to an epoch and vice-versa.

X-Delete-After is a convenience header allowing you to give the number of seconds from now when you want the object deleted. Swift will use this value to calculate an epoch time this number of seconds into the future and set an X-Delete-At header on the object.

# delete the object on Sat, 19 Dec 2015 19:18:52 GMT
$ curl -X POST -H 'X-Auth-Token: xxx' -H 'X-Delete-On: 1450552732' http://swift.17od.com/holiday-pics/sunset_glow.jpg

# delete the object 24 hours from now, 24*60*60
$ curl -X POST -H 'X-Auth-Token: xxx' -H 'X-Delete-After: 86400' http://swift.17od.com/holiday-pics/sunset_glow.jpg

See Expiring Object Support in the developer documentation for more information.

Segmented Objects

Swift has an object size limit of 5Gb. Larger objects must be split up on the client side and the segments uploaded individually. A manifest object is then used to create a logical object built up from the segments.

It’s not just large objects that can be segmented. Any sized object can be broken up. Having said that I can’t think of a good reason why you’d want to do this?

On UNIX the split command can be used to split an object up into segments. There’s two ways of using split, either give it the number of segments or the size of the segments you want to split the object up into.

# split an object up into 5 segments
$ split -n 5 at_the_beach.mp4 at_the_beach.mp4-

# split an object up where each segment is at most 5GB
$ split -b 5G at_the_beach.mp4 at_the_beach.mp4-

In each case the result will be a number of files called at_the_beach.mp4-xx where xx is an alphabetically ordered sequence of characters, e.g. aa, ab, ac and so on. This ordering is important because when Swift rebuilds the object it sorts the segments by name before concatenating them. Each of these segments should be uploaded into the same container.

The manifest object is an object with no content. Instead it has the header X-Object-Manifest. The value of this header is the container name and common prefix of the segments making up the object. Assuming the segments were uploaded into a container called holiday-pics and the prefix was at_the_beach.mp4- the header value would be holiday-pics/at_the_beach.mp4-.

$ curl -X PUT -H 'X-Auth-Token: xxx' -H 'X-Object-Manifest: holiday-pics/at_the_beach.mp4-' http://swift.17od.com/holiday-pics/at_the_beach.mp4

A GET for the manifest object will return the reassembled source object. Swift will stream each segment in sequence so from the client side it will appear as one continuous object.

See Large Object Support in the developer documentation for more information.

Metadata

Accounts, Containers and Objects can all have custom metadata headers associated with them. These headers are simple name/value pairs. Custom headers are distinguished from system headers with a prefix, X-Account-Meta, X-Container-Meta or X-Object-Meta.

Headers can be set on an existing account, container or object using the POST method. Alternatively the headers can be set when the container or object is being created using PUT.

# Set a header on an existing object
$ curl -X POST -H 'X-Auth-Token: xxx' -H 'X-Object-Meta-Location: Dublin' http://swift.17od.com/holiday-pics/at_the_beach.mp4

# Set a header on a container when creating it
$ curl -X PUT -H 'X-Auth-Token: xxx' -H 'X-Container-Meta-Year: 2012' http://swift.17od.com/holiday-pics/

Metadata can be retrieved using the HEAD method.

$ curl -i -X HEAD -H 'X-Auth-Token: xxx' http://swift.17od.com/holiday-pics/
HTTP/1.1 204 No Content
Content-Length: 0
X-Container-Object-Count: 3
Accept-Ranges: bytes
X-Timestamp: 1355861803.81992
X-Container-Bytes-Used: 2
X-Container-Meta-Year: 2012
Content-Type: text/plain; charset=utf-8
Date: Wed, 19 Dec 2012 20:07:03 GMT

Permissions

Permissions in Swift are controlled at the container level. Only users classified as administrators can create containers. By default regular users can’t create or access containers.

Users can be given read or write permissions for all the objects in a container using the X-Container-Read and X-Container-Write headers.

# Give bob and alice read access to the holiday-pics container
$ curl -X POST -H 'X-Auth-Token: xxx' -H 'X-Container-Read: bob, alice' http://swift.17od.com/holiday-pics/

# Give bob write access to the holiday-pics container
$ curl -X POST -H 'X-Auth-Token: xxx' -H 'X-Container-Write: bob' http://swift.17od.com/holiday-pics/

For more information see the ACLs section of the developer docs.

Pseudo-Hierarchical Directories

While containers can be compared with regular filesystem directories, it’s not possible to nest them. A container with thousands of objects can become extremely difficult to navigate and manage. With that in mind Swift supports a feature called pseudo-hierarchical directories. These are directory structures derived from the names of objects themselves. For example, lets say we have a container with the following six objects,

photos/2010/001.jpg
photos/2011/001.jpg
photos/2011/002.jpg
photos/2012/001.jpg
photos/2012/002.jpg
photos/2012/003.jpg

By using a delimiter character Swift can be asked to list the objects as if they were in a directory structure similar to,

/photos/
|-- 2010
|   `-- 001.jpg
|-- 2011
|   |-- 001.jpg
|   `-- 002.jpg
`-- 2012
    |-- 001.jpg
    |-- 002.jpg
    `-- 003.jpg

For example,

$ curl -X GET -H 'X-Auth-Token: xxx' http://swift.17od.com/adrian/?delimiter=/
photos/

$ curl -X GET -H 'X-Auth-Token: xxx' "http://swift.17od.com/adrian/?delimiter=/&prefix=photos/"
photos/2010/
photos/2011/
photos/2012/

$ curl -X GET -H 'X-Auth-Token: xxx' "http://swift.17od.com/adrian/?delimiter=/&prefix=photos/2012/"
photos/2012/001.jpg
photos/2012/002.jpg
photos/2012/003.jpg

For more information see Pseudo-Hierarchical Folders/Directories in the Developer Guide.

Swift All in One

Like anything, the best way to learn more about Swift it to play around with it. Luckily that’s relatively easy thanks to Swift All in One, a set of instructions describing how to setup a fully functional Swift cluster on a single machine (ideally a VM).