Hi everyone,
The technology we will look at in this article is Simple Storage Service, a file area that AWS has brought to us, where we can store the files of our projects, allow file versioning, and keep data safely.
What is Simple Storage Service(S3)?
The primary building blocks of a project are the storage of our project data. S3, which provides integration with our projects, along with many criteria such as how the data will be stored, who can access this data, security criteria for safekeeping, keeping the file encrypted, backed up, and historical version, also protects us from problems related to the OS layer while installing these systems.
Simply put, S3 provides you with a remote file server. Your concern is integrating this file server into your project and holding AWS accountable for all the rest.
Security
When we create an S3 for our project, we make this S3 bucket accessible to the user who created it and the users specified with the IAM Policy privileges allowed by this user (or root user). In addition, if we want, we can also provide access management by defining permissions to each file or bucket separately. Again, we can parametrically decide which users can access the files and how often. Thanks to its automatic encryption, we can be sure of the security of our data. In addition, monitoring processes such as who accessed the s3 bucket and when, which files were uploaded/changed by whom is one of the features that come with S3.
Scalability
If it is about storing data, one of the first problems that come to mind is “if the storage space I have allocated one day is insufficient, how can I expand this space without stopping the system?”. S3 provides almost unlimited space in this regard, allowing data to be stored without any limit, with the principle of “pay as you go”.
Let’s say you have a project and it suddenly exploded on social media, hundreds of people started signing up with the app, and an incredible load of files came in. S3 manages this load very comfortably, allowing us to move forward in a controlled manner.
Durability
Losing data is probably the last thing every company/project would want. The durability at the S3’s default offered level is 99.9999999%. If I give an example to imagine this value, I think it will be easier to visualize;
If we store a hundred billion objects in S3, we will lose at most 1 file.
Low-Cost Policy
When we start to talk about such features, the first thing that comes to mind in technology is “everything is fine, but how much more will I have to pay for so many features?” thought. However, within the S3 budget, there is a policy with different prices depending on how often you will call the data, how this data will be stored, how it will be backed up, and how it will be protected, and each has a significant price advantage compared to its alternatives. Pricing policy (as of October 2022) has varying prices between $0.022 / GB and ~$0.0125 / GB. In the “Glacier” type, which we can see when we go into the details, the price becomes more affordable and decreases to ~$0.004 / GB.
Ease of use
When it comes to storing data, we may all think of a cold environment filled with server rooms, air conditioners, and SSDs in these rooms, but you don’t have to deal with any of these to manage/use your hundreds of terabytes of data with S3. It is possible to manage your data in 3 ways;
- GUI (Graphical User Interface)
You can control it with the AWS Management Console, web apps, and mobile apps where each task can be executed with a single click or a tap. - CLI (Command Line Interface):
AWS provides the command line program suite to execute our tasks by running commands directly. - API (Application Program Interface):
AWS provides APIs that allow companies to quickly integrate the platform with other technologies in use.
Summary
- A fully managed and object-based
- One of the most common data deployment methods
- An infrastructure that can handle many scenarios
- Integrates with many other AWS services
- Highly Available and Durable
- Cost-effective
- Easily scalable and accessible
- (almost) Unlimited storage
- Extremely scalable
- File size support up to 0 bytes as small as 5 terabytes
- A structure that allows the file to be stored in different AZs and to be decided at the time of creation.
- Allows easy copying and/or duplication of the file to other zones
- Hosting all kinds of data extensions (video, music, audio, text and many more)
- 99.99999999999% durability
“Why AWS S3?” Now that we have the answer to the question, we can begin how we can use S3 with Django.
Setting up Amazon S3 Environment
We login to AWS and open S3;
In the standard of many projects, S3 is positioned so that it cannot be accessed from the outside, so it is not accessible from the outside. In our example, we activate the outside access as we will access it directly;
If we want our files to be versioned (not in the scope of the example, but if you need it in your project) or if we want our data to be encrypted, we need to activate it from the options section here, we can see the settings here, even if it is not in our example, it is possible to change these settings later, but after adding a few more steps will be necessary. , if possible, this decision should have been made when creating the s3 bucket;
Let’s create our S3 Bucket;
Configuring IAM Access Settings
Now, let’s start setting up the user settings to be able to access the s3 Bucket we created via Django. The IAM structure basically consists of 2 building blocks. We define a user that we will create for our application and the authorization groups that we determine the authorizations that this user will have and the authorizations that this group will have. First, let’s define the privileges we want to define and the group that will have these privileges;
In order to keep the example simple, we will add the default s3 access policy structure that gives access to all s3 buckets, but if you want to limit it, it is of course possible to limit it over the ARN information by saying “create policy”;
As an example, it is possible to create a group for customized access, thanks to this policy, we only grant access to the s3 bucket that you specify;
Now that we have created the authorization group and policy information we have created, we can start creating our user;
Authorized Programmatic User Creation
We’ve created our user group, assigned their privileges, now we’ll create a username and password that Django can use to access and communicate with S3 Bucket, return to the IAM page, create a new user from the user tab;
What we need to pay attention to here is that we choose “Programmatic access” when creating our user, that is, we are telling AWS that this user and my application will communicate with AWS, so an Access Key and secret are required. In the next step, we assign the custom authorization (or the default s3FullAccess) that we created in the previous section to our user;
In the last step, we check the privileges of our user;
After creating our user, we note the information on the page that appears (we will use it in our application):
Creating the Django Project
Now that everything is ready on the AWS side, we can start designing our Django application. I’m going through the standard installation steps, if you’re missing those steps, you can check here. It is critical for us to do the s3 integration of our project, to install the “boto3” and “django-storages” packages we need;
pip install boto3 django-storages
After the installation is complete, we show Django the Storager we want to use and add my configuration information:
settings.py
INSTALLED_APPS = [
'django.contrib.admin',
...
'django.contrib.staticfiles',
'aws_sample.apps.AwsSampleConfig', # uygulamamizin ayarlari
'storages', # AWS konfigurasyonları için varsayılan app
]
AWS_ACCESS_KEY_ID = '#######'
AWS_SECRET_ACCESS_KEY = '######'
AWS_STORAGE_BUCKET_NAME = 'django-sample-storage'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_STATIC_LOCATION = 'static'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_STATIC_LOCATION)
If all is well, when we issue the collectstatic command, our files should be uploaded to s3 Bucket under “static”;
Let’s test it, when we go to the http://127.0.0.1:8000/admin/ page, we can see that the CSS design has arrived if our files are correctly transferred to the s3 bucket. By viewing the page source we can see that these files are served by s3;
You can find the source code of the project here.
See you in my next post 🙂
Sources:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
https://www.educative.io/courses/detailed-workings-aws-s3/qZ4ApmolOr3
https://www.blazeclan.com/blog/5-key-benefits-of-amazon-s3/
https://www.quora.com/What-are-the-benefits-of-using-Amazon-S3
https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/
https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html
https://medium.com/analytics-vidhya/file-uploads-with-amazon-s3-in-a-django-project-54fb989e992d
