Goal

Figure out how to store images on a private AWS bucket.

Notes

Before considering how to Django Bulk Upload cosmetics into the app, it makes more sense to figure out how to load them into a secured S3 bucket. I've used this technique before for the DIMMiN App, but I'd like to make some adjustments that I glossed over the first time when constructing DIMMiN.

I consulted with ChatGPT to get its thoughts on how to configure this setup and it was quite helpful. I already had the Environment Variables configured for file uploads so that was already handled. I also already had boto3 django-storages installed because requirements were already installed when I ran

pip install boto3 django-storages

I created the following configuration for Media Files in my hithub/settings.py file:

### AWS S3 FOR MEDIA STORAGE
# AWS S3 for additional storage with larger files (audio, bulk images, videos)
AWS_ACCESS_KEY_ID = env.str("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = env.str("AWS_SECRET_ACCESS_KEY")
AWS_STORAGE_BUCKET_NAME = env.str("AWS_STORAGE_BUCKET_NAME")
AWS_S3_REGION_NAME = env.str("AWS_S3_REGION_NAME")
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_DEFAULT_ACL = 'private'
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
AWS_QUERYSTRING_AUTH = True  # to ensure that files are not publicly accessible

# For media files
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/media/'

I'll also have to remember to set these Environment Variables using the Heroku CLI in Production.

I tried to upload a test cosmetic to the S3 Server but was denied (as expected, I had a feeling there's something missing here). The error was an AccessDenied error when trying to POST media to the S3 bucket. The exact details of the error were:

# ClientError at /admin/fighters/cosmetic/add/
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

ChatGPT mentioned that this is likely an issue with my IAM User permissions configured to my S3 bucket. I'll need to allow my app to have the following permissions:

• PutObject (POST)
• GetObject (GET)
• DeleteObject (Delete)

Since I had encountered this kind of error before I figured it would be worth it to understand boto3 a little better. I decided to spin up a shell with the

python manage.py shell

command and follow along with a little bit of the Bucket Policy Documentation. I ran the following code in the shell:

import boto3

# Retrieve the policy of the specified bucket
s3 = boto3.client('s3')
result = s3.get_bucket_policy(Bucket='BUCKET_NAME')
print(result['Policy'])

but received a response which declared that

An error occurred (AllAccessDisabled) when calling the GetBucketPolicy operation: All access to this object has been disabled

I decided to play around with the IAM User settings and added in Multi-Factor Authentication to my account while I was at it. I found that I was the only user in the system (good!) and I had root access (as expected).

I saw two potential security configurations I may have to manage,

• S3 Access Grant and
• S3 Access Point

It looks like S3 Access Grant are more for external sharing of data while S3 Access Point are better geared towards app-level permissions ( GET / POST etc.). It looks like I'll be more interested in setting an S3 Access Point to help the HitHub App access its files.

I decided to allow this point to be accessed via the internet instead of exclusively from the Virtual Private Cloud and set up the AWS Access Point ARN to allow ease of access to this policy. In the future I may configure a VPC but for now I want to make sure I can get things up and running.

After creating the S3 Access Point I found out that it still was unable to serve images because the overall permissions were blocked in the Permissions Overview section:

Access
Bucket and objects not public

I decided to remove the S3 Access Point and configure the CORS Permissions directly on the S3 Bucket itself. Unfortunately this was also blocked because my

bucket policy grants a level of public access that conflicts with your Block Public Access settings.

It looks like I'll have to create a new IAM User for my HitHub App to allow it specific access to the S3 bucket. I gave the HitHub App all permissions to the hithub S3 bucket. Unfortunately this still didn't work. I'll have to figure out in more detail how to configure these bucket permissions tomorrow.

Results

• Learned about different AWS S3 Permissions required to access a private S3 Bucket.

Next Time

• Figure out the correct configuration that allows my HitHub App to access data in an S3 Bucket
  • Then make sure to properly configure the Media Files in Django to handle GET and POST requests for image files so we can start uploading Cosmetics for our Fighters.