Goal

To learn how to bulk upload to my Django database

Notes

Working on the HitHub App while streaming. I decided to add in first_name, last_name, and nickname to the SpiritFighter class of the fighters/models.py file.

I consolidated the data for the Spirit Fighters into a JSON file via the Documents/HitHub/fights_simulator.ipynb file.

I also learned how to bulk upload records via the Django Bulk Upload operation. I was able to access the django python shell first by running the

python manage.py shell

command. Then, I could access the SpiritFighter class via

from fighters.models import SpiritFighter.

Then, I refreshed the SpiritFighter data. I ended up writing the following script to execute in the shell:

from fighters.models import SpiritFighter
import json
from itertools import islice

with open(r"C:\Users\Billy\Documents\HitHub\bulk_uploads\Fighters\spirit_fighter_metadata.json", "r") as f:
    spirit_fighter_data = json.load(f)



# Build up the objects data
objs = []
for fighter_id, fighter_data in spirit_fighter_data.items():
    objs.append(SpiritFighter(
        ufc_id = fighter_id,
        height = fighter_data["height"],
        weight = fighter_data["weight"],
        stance = fighter_data["stance"],
        first_name = fighter_data["first_name"],
        last_name = fighter_data["last_name"],
        nickname = fighter_data["nickname"],
        date_of_birth = fighter_data["DOB"],
        w = fighter_data["w"],
        l = fighter_data["l"],
        d = fighter_data["d"],
        stats = fighter_data["stats"],
    ))

batch_size = 100
while True:
    batch = list(islice(objs, batch_size))
    if not batch:
        break
    SpiritFighter.objects.bulk_create(batch, batch_size)

Unfortunately, I'm still getting the following error:

django.db.utils.IntegrityError: UNIQUE constraint failed: fighters_spiritfighter.id

Which is bizarre considering that the ID should be auto-incremented. I can see that the first 100 SpiritFighters have been loaded in successfully, so I wonder if it's an Asynchronous loading thing? Either way it's a great opportunity to try out Django QuerySet filters.

I'd like to update the script to Filter the uploads to only those that do not already exist.

Results

• Learned how to Django Bulk Upload
• Finished putting together the SpiritFighter model of the Fighters App

Next Time

• Learn how to upload more than 100 records via the shell