My Projects

An FPV quadcopter post-flight telemetry visualization tool

• Python, Streamlit
• Docker, ECR, ECS
• Terraform, Terraform Cloud
• Git, GitHub, Github Actions

Streamlit application, authored in Python, that transforms an uploaded CSV file and displays flight metrics, a map with a flight path trace, and graphs of relevant sets of data. The app and its dependencies are packaged as a Docker image, uploaded to Amazon Elastic Container Registry, and launched serverlessly using AWS Fargate.

The Terraform configuration for this project is modularized, and several files are provided as templates for manipulation within GitHub Actions. The primary Actions (both for building and uploading the app's Docker image, and for deploying the app's infrastructure) are reusable, and can be called from either the dev of main branches of the repository.

Some of the challenges I solved during development:

• Many of Streamlit's abstracted integrations (for example, plotly) are easy to work with, but I found adding a map with a trace of the quadcopter's GPS flight path to be rather difficult. After a lot of trial and error, I determined the format that the flight path map layer needed and added a transform to the Pandas dataframe to aggregate the appropriate columns.
• Keeping the Mapbox API key secret took me down several rabbit holes, because the Streamlit app looks for that key in a .toml file. Since TOML is a markup language and not a scripting language, I couldn't simply pass the key in as an environment variable. My solution was to create the necessary config file as an early step in the GitHub Action, using the 'sed' command, before building the Docker image.
• I set up two separate AWS accounts within an organization for this app: one account for development and testing, and one for production. I wanted a way to deploy to the correct account depending on which branch of the repository was being used, so that I didn't end up with multiple infrastructures within one account. After testing each action in the dev account, I refactored them to be reusable with certain required secrets and parameters. These reusable Action scripts are called by smaller, branch- and path-dependent Actions.

Design narratives on Dev.to (click to read):

• Introducing the Skyboy App
• CI/CD: GitHub Actions with Docker and Amazon ECR
• CI/CD: Branch-based Terraform Deployment

To-dos include: checks and conditionals for different CSV column headings, to accommodate different radio/flight controller combinations; automated unit-testing; ability for users to register/authenticate and upload telemetry files to S3.

Automated ETL processing pipeline for COVID-19 data

• Python, boto3 SDK, Pycharm unit testing
• Lambda, DynamoDB, SNS, EC2
• Terraform, Terraform Cloud
• Git, GitHub, Github Actions
• Redash

An AWS Lambda function, authored in Python and using boto3 to interact with other AWS resources, downloads data related to COVID-19 and performs an ETL process. The data is written to DynamoDB, where it can be queried by a self-hosted instance of Redash to create a visualization dashboard. A message is sent to a Simple Notification Service topic whenever the process finishes or encounters an error. The Lambda function is triggered daily by an Amazon EventBridge rule.

I am particularly pleased with the choice I made to provision all of my infrastructure using Terraform. This was my first time using it, so I spent several hours learning the basics and applying that knowledge to my configuration. Terraform seems like a great tool to use in my pipelines going forward, especially if/as I branch out to other cloud providers.

Numerous challenges presented themselves throughout the development process. Some examples:

• After working out the Python logic locally, I used multiple AWS accounts for testing and production. Each account's resources were deployed from a separate repository, which made for a lot of copy/paste. This process could be streamlined using git branches and environmental variables, something I would like to learn and practice.
• An IAM permission scoping issue threw me for a loop, sidetracking the project for several hours as I erroneously refactored the Terraform configuration.
• There were several key parts to this challenge that I had to (or wanted to) learn from scratch: boto3, AWS's Python SDK; Terraform; and unit testing in Python. Had these topics been familiar to me already, I suspect this project would have taken far less time.

Future improvements may include: refactoring the Terraform configuration file; writing more comprehensive unit tests; and revisiting queries and visualizations in Redash.

NB: this application has been refactored into a static website to save costs and is no longer deployed on Elastic Beanstalk. The source code can still be found in this GitHub repository.

A Flask web application with Bootstrap CSS

• Python
• Flask, SQLAlchemy, smtplib, Werkzeug
• HTML, Bootstrap CSS
• Git, GitHub, Github Actions
• Amazon Web Services, AWS Elastic Beanstalk

This webapp was created as a capstone project for the Python Bootcamp taught by Angela Yu of The App Brewery. Some backend Flask code was written by me for an earlier project, while the frontend code is almost exclusively written from scratch. It is deployed on AWS Elastic Beanstalk via GitHub.

I am particularly pleased with two aspects of this application: styling the HTML with Bootstrap, a framework with which I had little experience; and figuring out the deployment process, which included using the AWS EB CLI tool locally, associating the application with a MySQL RDS instance, and setting up a GitHub CI/CD pipeline to automatically push changes to Elastic Beanstalk.

Examples of the many challenges this application presented me:

• Proper form validation: I decided not to use wtf.quick_form() to create the various HTML forms, so the validate_on_submit() method was invalid. Documentation led me to an alternate method for validating form fields.
• Custom CSS: I wanted to define my own color for the Bootstrap buttons, but importing the Bootstrap CSS via a CDN meant that I had no easy way to manipulate built-in values. A good deal of searching led me to the code I needed to calculate color hex values and create my own btn-custom class.
• SMTP: my web application was unable to send emails (from the Contact page) after being deployed onto AWS. A trip into my EB environment's EC2 logs first revealed a CSRF error, so I explicitly imported CSRFProtect from Flask-WTF. Although that hurdle was cleared, POST requests continued to time out. Several searches on stackoverflow eventually revealed that AWS throttles outbound EC2 traffic on port 25. Explicitly defining a different port for SMTP traffic has restored email functionality.

Future improvements may include: figuring out AJAX requests so that this webapp can be served through CloudFront (caching and CSRF don't play nicely); defining Elastic Beanstalk configuration settings with code, instead of manipulating the EB environment in the AWS console; improved blog formatting and pagination; more error-handling; some sort of alert (email?) system when users comment on a blog post.

A serverless cloud resume website

• HTML, CSS, Javascript
• Git, GitHub, GitHub Actions
• Amazon Web Services, AWS Serverless Application Model, AWS SAM CLI
• Lambda, DynamoDB, API Gateway, S3, Route53, Certificate Manager

I completed this project shortly after obtaining the AWS Solutions Architect Associate certification and just before I began an online bootcamp to learn Python. As I didn't know much code at that point, I chose to use a pre-built HTML/CSS template for the frontend. Loading the website triggers an AWS Lambda function that increases the visitor count in a DynamoDB table by one and returns the new count to the website.

This project contained many firsts for me, including implementation of a CI/CD pipeline, use of the Serverless Application Model CLI tool, and creation of an AWS Lambda function. I spent numerous research hours online as I worked towards completion of the website and its backend counter. I attempted to write the Lambda code myself, but ended up referencing several other similar functions to piece together a working solution.

Since the completion of this project, I have learned quite a bit more about coding and about the way these services function and work together. It would be an interesting exercise to see if I could author the Lambda function on my own at this point.

An API-based Python GUI app

• Python
• guizero, tkinter, requests

After working with the Pixe.la APIs through the console during a bootcamp project, I decided to create a graphical user interface application that would allow me to track my time spent coding and learning on a pixel graph. I reused the API-related code that I had authored during the bootcamp project and built the GUI from scratch.

This was my first coding side project, written over the course of a week as I moved on to other Python lessons. I am particularly pleased with my implementation of guizero, a wrapper for tkinter that I had not used before. I relied heavily on the guizero documentation for reference.

Some notable challenges:

• Image resizing: I wanted to include the Pixe.la logo as part of the app's header, but when I defined the desired width and height of the image, only a small corner of the full image was shown in the app window. Some searching led me to discover that guizero required the Pillow library to scale images. Installing the Pillow package created the desired outcome.
• Error handling: the Pixe.la API requires the 'date' header to follow a specific format, so I needed to both convert the date string entered into the app, and catch other errors before data was passed to the API. I learned how to use a try/except block to catch invalid entries, and used the datetime module and strftime to convert the date entry into an appropriate date object.
• GUI operation: I wanted the app to have a reset button to clear text boxes and reset radio button selections. Research revealed that guizero did not include wrappers for this type of functionality, but that I could use tkinter methods with guizero to solve this challenge. In the case of resetting radio buttons, this involved creating a list of the options and calling tk.select() on the desired list index.
• Including the graph: Pixe.la graphs are rendered in SVG format in a browser window. I attempted to figure out how to display a graph in the GUI window, only to discover that there doesn't seem to be a Python library that can handle SVG code effectively without relying upon a separate image processing application. I did not like the idea of having to include a separate program with this app, so I settled for a button that opens a link to the selected graph in a new browser tab.

For the time being, this application is complete. Future improvements may include: creating an executable using PyInstaller; adding the ability to register new users and create new graphs; displaying a graph in the app window if a library for doing so becomes available.