Object Storage
Summary
What is object storage? A very simple means to store simple objects. An object is closer to a filesystem than a database. In most implementations there are two concepts to familiarize yourself with. The first concept is generally called a Bucket. Much like it sounds, a Bucket is a receptacle use to store items inside of it. These items are generally called an Object. The purpose of a Bucket is to store your objects in an isolated and secure way. The Object and its Key can be fairly large in comparison to a traditional filesystem.
A Bucket generally only has objects, not folders or directories.
The Key
A Key may generally be any UTF-8 characters and can even be used to virtually represent a hierarchical data structure. An Object is referenced by its Key. You may associate custom metadata with an Object. However, many implementations provide permissions that may be bound to a Key or part of one.
The Object
This metadata is generally not useful for identifying objects rather providing additional information about the Object itself. An Object may be very large, a single object may be several Terabytes (TB).
Persistance
The major purpose of an object store is for persistence. Generally many different storage classes are available that vary in performance and price. In most circumstances you may a fee for the data stored per month. Depending on the implementation you will also pay for data transfer.
Use Cases
The general point of object storage is a very simple hassle free way to store any data. You don’t need to keep infrastructure laying around just to store and retrieve your data. In the cloud computing world the notion of cloud storage was one of the earlier services available.
Backups
This provided companies with easy backup solutions. No messy physical drives or software.
Static web sites
The ability to make your object storage data accessible via a web browser added an additional dimension of functionality. Not only can you store your files on the cloud with great ease, but you can also serve static content at the same time. The ability to easily create static websites that did not require provisioning of any infrastructure is an incredible service.
Simple key value stores
I’ve used object stores as very simple key/value systems for various needs in creating proof of concepts. A fairly simple example is for storing security tokens for a specific identity. Let’s say your users have a 1024 byte token that they get after they login. A list of the user’s permissions are encoded into a JSON document and stored as the value alongside the user’s token. When the user presents their token the security system will obtain to fetch that entry from the pre-defined Bucket. Should no object be present then the token is invalid. Otherwise the value will be treated as the user’s permissions. I’ve used this sort of approach for very simple security systems that don’t really need a database. This provides a very easy way to pre-populate the object store with valid security tokens. This is much cheaper simpler than provisioning an in-memory object store. That is obviously better, but you will pay for that speed.
User’s storage
I’ve had systems that allowed user’s to upload their own files to be used in the system. Object storage with the username in the hierarchy provides a very efficient way to have a hassle free way to store user supplied data and not worrying about any access.
Work queue
For data like image processing, log processing, or really any data analytics. Many object storage providers provide subscription mechanisms to dynamically respond to objects and how they interact with the system. I’ve used this approach for processing logs dynamically.