created: 2024-12-11 06:16 tags: - daily-notes

Wednesday, December 11, 2024

<< Timestamps/2024/12-December/2024-12-10-Tuesday|Yesterday | Timestamps/2024/12-December/2024-12-12-Thursday|Tomorrow >>

🎯 Goal

• [ ] Automate the ETL pipeline to go from a Local Version of my notes to executing one command that updates the Production version of my Obsidian notes

🌟 Results

• Started automating the ETL pipeline to take my Obsidian notes from a Local Version to the Production version of the BigBrain App
• Developed a script that could upload my local Markdown files (and their original Metadata) to an S3 bucket for a given Vault

🌱 Next Time

• Create a Django Management Command that can load data from Markdown files directly from an S3 Bucket into the BigBrain App.
• Potentially create a scheduled CRON Job with a Celery worker so that updates to these notes are periodically refreshed
• Update the orphan-handling component of the S3 uploading Django Management Command to delete files that are no longer present.
• Start working on a devlog page where I can publish my notes from my DIMMiN App development (hello!). Maybe place this under the Blog App section?

📝 Notes

First thing I want to do is try to push my notes to Production manually. I updated my DIMMiN App with

git push heroku master

to push the new changes to the BigBrain App to Production, then applied Database Migrations via

heroku run bash -a dimmin
python manage.py migrate

Next I built out a Django Management Command to apps\bigbrain\management\commands\upload_obsidian_vault_to_s3.py that can take a Local Version of my Obsidian notes and upload the Markdown files to an S3 Bucket. I explicitly specified the Metadata for creation and modification dates to match that of my local vault. It's pretty inefficient because it just uploads all the files every time, so I put a check for only files whose modification time has changed. However it's still not as efficient as I'd like it to be as it makes a call for the Metadata of every single file (which adds up to a lot of requests). I wonder if there is a way to bulk pull all of the metadata for the files in one request... I ended up using an approach that sent batch requests to check the metadata for all the files which was much faster. Now I can upload any Obsidian vault from a Local Version to an S3 Bucket with

python manage.py upload_obsidian_vault_to_s3 --vault-path "/path/to/vault"

Next I'll want to make a variant of the upload_obsidian_vault Django Management Command that I developed a couple days ago to import directly from S3. This could handle the Signed URL logic as well which could kill two birds with one stone.

It may also make sense to develop a scheduled command (CRON Job from a Celery Worker) that pulls the data directly from S3 into the BigBrain App at an interval so that I don't have to . Maybe I could also keep it as a Django Management Command so that I can manually apply it as well.

Notes created today

List FROM "" WHERE file.cday = date("2024-12-11") SORT file.ctime asc

Notes last touched today

List FROM "" WHERE file.mday = date("2024-12-11") SORT file.mtime asc

(Template referenced from Dann Berg, can be found here)