On the third day of my career, I was helping Netflix restart their Apache Cassandra cluster as employee #9 of a geo-distributed company that grew to 440 employees 4.5 years later. Later in my career, I helped small businesses decommission half their nodes while tripling performance, stabilized large clusters of 800+ nodes, and worked closely with and for companies like T-Mobile and Adobe. Throughout my career, I've always enjoyed facing the challenges that accompany distributed systems at scale.
I first heard about Bitcoin when 1 BTC was worth $0.99. When it retraced from $320 to $76, I picked up 14.7 BTC from Mt. Gox by wire transferring $1,200 to BitInstant through a dirty red phone at a Walgreens at 3 AM in 2012. That week, I read Satoshi Nakamoto's whitepaper 3 times.
Having grown up reading Wired magazine, I have always been an enthusiast of new technologies and now held something of potential value. I kept up with the crypto space, hosted local educational sessions, attended the 2015 Texas Bitcoin Conference, and visited a few Meetups. In 2018, I liquidated all of my cryptocurrency, including my remaining 5.41 BTC, to place a 20% downpayment on our home to avoid paying private mortgage insurance (PMI).
After a brief disconnect from the blockchain community, my interest in the community returned after the return of the defi (Decentralized Finance) buzz and before the popularity of NFTs (Non-Fungible Tokens). Shortly after, I left Adobe to join Audius, a company rebuilding something akin to Soundcloud on the Ethereum, Solana, and POA blockchains.
Growing up, I wanted to be Eddie Murphy in Beverly Hills Cop. For several reasons, that very specific situation is not possible. My love of film has followed me from my hometown cinema, Cinema III, through my double major at The University of Texas at Austin, and into the rise of this generation of AR/VR/360 Video technologies. Now that Generative AI is making its way into the generative video space, I'm excited to see how this new technology will help shape creative ideas into releasable content.
For over a decade, I've enjoyed being at small, nimble companies since it feels like every small change is your fault. Whether it's a mistake or a success, the company is affected because of you. If you need help building something powerful, have a collaborative and intelligent team, and like the experience I could bring to your project, I'd love to chat. Please reach out.
Own the developer experience and release pipeline from within the Foundations team. Supplement the Engineering team's knowledge of Docker Compose container-based services, Kubernetes namespace management, Helm deployment charts, Terraform-backed infrastructure, CircleCI deployment workflows, Grafana monitoring and alerting dashboards, OpenSearch event logging, and PostgreSQL database administration via live Confluence wikis. Standardize test data generation, deliver test stability, and harden workflows for local development and cloud deployments to AWS-hosted, Python-based environments.
Reduce local build times by 70% within 2 weeks of joining.
I was hired as a consultant for AxonOps to performance test the AxonOps Agent, publish a blog post, and present my findings with a co-founder at the AI.dev Open Source GenAI & ML Summit / Cassandra Summit 2023. I co-led four scheduled pre-conference training sessions, offered technical demos, and provided booth support at the 2-day summit.
Follow up blog posts and videowork to follow.
At Audius, I contributed to the blockchain-powered infrastructure through various improvements on the discovery nodes, which provide blockchain indexing and search functionality to the network, apps, and website.
The 119-node network hosted by 23 public service providers serviced 283M+ requests and 5M+ unique users monthly. Additionally, I heavily contributed to the Docker Compose project to achieve closer parity between local and production environments.
My first task was to create and run the first load tests on the Audius network. I wrote the tests in JavaScript and generated blockchain events on POA's testnet to guarantee the mainnet could sustain 6-months of projected growth. I pushed the code to a private repo to mitigate potential DDoS attacks.
The next effort was to add observability to the project by adding centralized logging and metrics. Since a few alerts existed in Pingdom, we began with the centralized logging project to reduce the need for production access.
I quickly set up centralized logging with ElasticSearch, Metricbeat/Filebeat, and Kibana hosted by Elastic Cloud for development, staging, and production docker containers.
I then added self-hosted Prometheus and Grafana dashboards for network, governance, and blockchain transaction metrics. I added alerts for various on-chain activities like interactions with our governance contract and transactions from treasury wallets.
I also created Grafana alerts to deploy reward-slashing governance proposals, via CircleCI pipelines, for service providers who weren't adhering to the upgrade window.
I also designed and automated the build, test, staging, canary, and release pipelines using CircleCI. Included within the release pipelines was a registration system designed to allow developers to reserve properly tagged instances within the Staging environment without fear of the release pipelines interacting with reserved nodes.
The last steps in the release process included automated smart contract creation and post-timelock execution, as well as Discord and Slack announcements to gather proposal votes and remind service providers to upgrade.
In 2018, Adobe acquired Magento, a SaaS platform previously owned by eBay and primarily based in Barcelona, Spain. I was hired by the former Director of Security at Umbel to ensure the databases could scale to meet the seasonal needs of their incoming enterprise-level customers.
I worked on the Inventory Service, which was being rewritten into Golang microservices while the PHP monolith's inventory service was in maintenance mode. Shortly after becoming comfortable with Golang and three months before Black Friday, our DevOps team member in Barcelona gave a 30-day notice before transferring to a different department in the US.
In under a month of remotely shadowing my team member in his local time zone, I quickly got up to speed on GitOps methodologies and Kubernetes release management for staging, pre-production, and multi-region production environments. New Git commits would trigger my newly rebuilt Jenkins X pipelines that maintained healthy, continuous delivery across our environments. The infrastructure and release process were validated over a 3-day engagement with AWS Solution Architects.
As we were getting close to our beta release of the new Inventory Service, Adobe announced its latest offering, Adobe Commerce. Adobe Commerce would be written from scratch in Java following Adobe's company-wide infrastructure policies.
For the rest of my tenure at Adobe, I created a comprehensive wiki for the Sourcing Engine and Configuration System as we transitioned ownership to contractors in India tasked with sunsetting the platform. The wiki included complete documentation of technical processes, troubleshooting steps, and the new release management orchestration I redesigned to simplify the documentation and handover.
Our team was the first out of five to entirely hand over our project due to our robust test cases, organized logic, and active use of a wiki, even though we only shared a 3-hour workday overlap.
During periods of low customer engagement at The Last Pickle, I often engaged with T-Mobile's Cassandra teams. Questions around monitoring details and alert resolution were common across Developer teams and the broader Infrastructure team. When T-Mobile chose an org-level Grafana instance to monitor their applications and infrastructure, they asked TLP to create Cassandra-centric dashboards that could be used across their teams.
A coworker and I delivered domain-specific dashboards separated into clearly labeled rows containing full runbooks in all important panels' tooltips. We also configured alerts with detailed resolution guides and matching threshold explanations in each alert message. These dashboards were later released as part of a Docker Compose project during a presentation at the Austin Cassandra Meetup and in a follow-up blog post.
The same dashboards were slightly repurposed as DataDog's official Cassandra dashboards, which I presented during DataDog Summit 2017. My coworker, Alain Rodriquez, published a blog post on DataDog's blog shortly after the summit to announce the dashboards we worked on.
During an engagement with a new inbound customer late on a Thursday, I diagnosed and repaired an 800-node multi-region database cluster experiencing split-brain networking issues where about five versions of the schema were simultaneously live causing a percentage of failed read and write requests.
After the CEO and Principal Architect diagnosed their issues for the better half of an afternoon and got ready to sign off, I joined and took the lead on the call. Within 30 minutes of listening to their symptoms, we began upgrading off JDK 1.7.0, synced NTP clocks with the same time server pool, and set alerts for duplicate schemas, resolving their issue in time for a slow rollout over the weekend.
Over three months, we delivered heavily detailed usage, data model, and operations runbooks and helped them implement most solutions. These runbooks contained debugging guides, indicators of potential high load scenarios, ideal metric thresholds, and actionable early warning and alert messages. From this collection of runbooks, we distilled two commonly seen topics into quick blog posts for the community: Re-Bootstrapping Without Bootstrapping (02 Aug 2018) and Assassinate - A Command of Last Resort within Apache Cassandra (18 Sep 2018).
After repeatedly asking for the same information across customers of various cluster sizes, I began automating my metrics collections using a simple script. This script was then productized as the goto method for collecting diagnostics for any new clients. It was soon used as a snapshot when our customers were experiencing issues and before making major operational changes.
A companion script was quickly added to automate the markdown generation of most runbook sections, including extracted values, command line output snippets, templatized data modelng sections, and personalized sections depending on cluster health.
The collection script and diagnostics generator were productized at The Last Pickle and transferred to DataStax after they acqui-hired the team. This tooling is still used by the Support team at DataStax in 2024.
I also contributed to Sandwich, an internal project that used Apache Cassandra, Apache Kafka, AWS Lambda, and upgraded versions of my templatized runbooks to quickly analyze Cassandra clusters and deliver personalized runbooks to customers ahead of the most common potential issues. Unfortunately, work on this product was not continued after the acquisition.
The Last Pickle was comprised of a global team of 13 based out of New Zealand with consultants in Australia, France, the Canary Islands, Austin, Denver, and California, I quickly learned how to be an effective team member across the International Date Line since I was the first engineer within the US time zones.
Locally, I continued to be involved in the Cassandra community by attending and presenting at Meetups, contributing asynchronous request pipelines to the DataStax Python Driver for Apache Cassandra, speaking at DataDay 2018, and helping the Cassandra community on Freenode (the Internet Relay Chat (IRC) network).
At Umbel, I was part of the three-person Architecture team consisting of the Chief Architect, the Senior Data Scientist, and myself. Under minimal guidance from the Chief Architect, I developed an asynchronous API written in Python which relied on Apache Cassandra, Amazon DynamoDB, and PostgreSQL that increased processing speed by more than 3x while reducing costs by 50%.
By the completion of the project, hourly jobs were completed in under 18 minutes at half the cost with reliable data replication and transparent denormalization.
These optimizations were made possible due to an asynchronous data object that handles denormalization, client-side retries, metrics, hydration, and transactions. The design heavily followed Volatility-based Decomposition planning and borrowed from Microservice, Event-Driven Design, Time Series, and Command Query Responsibility Segregation (CQRS) principles.
Before merging with MVPindex, Umbel was a data management platform focused on transforming the sports and entertainment industry's fan data into actionable insight to increase revenue. When I wasn't implementing best practices on the Cassandra cluster and effectively using connection pooling to improve job performance, I was instrumental in kickstarting Umbel Labs' innovation initiatives by initiating the first patent creation process, researching how best to use blockchain technologies like Enigma to transform Umbel into a "data bank," and experimenting with immersive reality to find new avenues for customer engagement during live sporting events.
During my time at DataStax, I presented at Meetups on topics ranging from cluster expansion, cluster management, baking auto-clustering Amazon Machine Images (AMIs), and using tools like the Cassandra Launcher which has upwards of 117k downloads on PyPI.
In addition to Meetups, I've given lightning talks at Cassandra Summits, presented at Data Day Texas, and answered questions on DataStax Support Forums, Planet Cassandra, and Freenode.
My blog posts on multi-datacenter replication (2012) and running Cassandra on EC2 (2014) are still live in 2024 due to the amount of traffic they continue to generate. The International Journal of Science and Research (IJSR) referenced my multi-datacenter blog post in 2013.
To meet the demands of being the primary support contact for 9 months as my manager was primarily at on-site engagements, I created internal automation, command-line tools to download, extract, and organize diagnostic logs and files, PagerDuty widgets for Zendesk with CLI support, and a Cassandra heapspace calculator that consumed diagnostic reports and the settings file.
After owning DataStax Support for 9 months, I trained 5 of the 6-person team, including the new manager, who were hired to replace me. Before on-ramping the distributed team, I was the person of contact for 200+ organizations, including 15 of the Fortune 100. The knowledge base, troubleshooting guides, and training material I wrote during that time period are still used in 2024.
As a Software Engineer in Test, I single-handedly wrote the Python and Java Drivers testing suite for Apache Cassandra, contributed to our internal cluster testing tooling, and transitioned our infrastructure from EC2 to RightScale. When I transitioned out of QA, I created a complete documentation set and trained 4 of the 5 replacement hires, including the new manager.
As part of the elite team of Solutions Architects, I listened to post-mortems and pain points that large corporations and financial institutions were experiencing with data storage and request throughputs at scale. I then turned these real-world scenarios into easy-to-understand marketing demos to showcase at conferences, including Azure's first tech conference, and through developer blog posts.
As a team, we could iterate on the overall objective of each new project quickly and I would be left to complete the project as they continued their heavy on-site schedules.
The demos I created merged into a single full-stack demo, operational training, and certification platform built with Python, Flask, Jinja2, Bootstrap, and Docker to run on EC2 and Azure instances.
Together with a colleague from the Training team, we administered DataStax Certified Partners training sessions using different parts of the platform for the course materials and class projects.
I was accepted into the Communications school at the Moody College of Communication and followed a post-production leaning in the Radio-Television-Film degree plan.
I was officially a Computer Science major in my third year after being turned away by the College of Natural Sciences registrar's office who advised me that being registered as a double major would raise my tuition. Since I was following the Bridging Disciplines Programs' Digital Art & Media interdisciplinary curriculum, I had access to courses that would usually be restricted to Computer Science majors. I continued to plan my course schedule until registration with the College of Natural Sciences was required.
I am still an active Advanced Communication Technologies Laboratory (ACTLab) community member which started as a spiritual sister program to Nicholas Negroponte's MIT Media Lab. Sandy Stone's program was born in Austin in 1993 and hosted the most active web server in Texas during the early days of the Internet.
The program continued to explore audio/visual projects on various topics until it spun out of the University of Texas at Austin in 2014 into an academic community network across multiple institutions.
Additionally, I was both a student and instructor at the Division of Instructional Innovation and Assessment's Advanced Digital Media Institute where we covered new media tools, like the Adobe product line, along with video production software, like Avid and Final Cut Pro, and 3D software, like Maya and Blender.
In high school, I experimented with Maxon Cinema 4D and later Maya in college. In 2007, I created my first 3D legal animations using Blender 2.42. Since then, I've produced a few animations, some publicly available. While the animations are court-ready, my high-impact visual graphics help present complex information clearly and memorably, leading to quicker resolutions and higher settlement amounts during mediation, allowing all parties involved to move on from the traumatic event.
I run a home Linux "server" using an 8-core, 16-thread CPU with 64 GB (max 128 GB) of memory and a 1GB GPU that I use as a centralized code repository that also runs Docker Compose services for Home Assistant, Mattermost, CUPS printing system, and other development containers and smaller scripts for things like running Cassandra and decrypting luks encrypted volumes.
For my rendering machine, which also houses my Steam library of unplayed games, I currently run Windows 10 Pro with a 4-core, 8-thread CPU with 24 GB (max 24 GB) of memory and an 8 GB GPU.
Both are projected via an Epson Pro G7400U, with 3LCD and e4k technology, and an aftermarket Short Throw Zoom Lens to fill as much of the 10'x8' screen as possible.
Audio is provided via in-wall wiring by the Logitech z906 5.1 Surround Sound Speaker System that is THX, Dolby Digital, and DTS certified. A Monolith by Monoprice THX AAA Balanced Headphone Amplifier powers AKG 240MKII (semi-open) and AKG K702 (fully-open, reference) headphones for immersive cinematic viewing with a warm soundstage and full lows.
I purchased all the equipment over five years through Slickdeals steals. My home cinema is lovingly called Cinema I, a throwback to my hometown's quaint Cinema III which contained three screens and housed many of my best childhood memories.
All other laptops I own are running Linux using the i3 tiling manager with synced dot-files that are MacOS compatible with additional support for yabai, a window management utility, and skhd, a hotkey daemon.
WTFDoIWat.ch is a small pandemic project that gives personal film recommendations with almost no plot discussion, a throwback to video store clerk recommendations. Very little was written, but I hope to revisit this project one day.
This project started in 2014 and focuses on 360 videos. Since 2014, I've collected an Entaniya Fisheye M12 280° lens, ten secondhand LG 360 cameras that shoot 2K video, a Panasonic LUMIX G85 camera which shoots 4K video, and an array of audio/video production equipment purchased through Slickdeals at discounted prices.
While I have a few ideas for individual scenes, I've yet to find a story compelling enough to tell. I have the bulk of the equipment ready to go and keep an eye on products like Topaz Labs and Stable Video Diffusion that may quickly help bridge the technological gap of the LG 360 compared to even something like the RICOH THETA Z1 or Insta360 Pro 2.
I've spoken on the theory of virtual reality and 360 films in my hometown at the Eagle Pass Independent School District's San Luis Elementary, Memorial Junior High, and CC Winn High School's Gifted and Talented groups, STEM classes, and La Frontera Film Festival. You can view a recording of the same slides and general content during a presentation at the 2018 ACTLab New Media Conference.
