UpKeep
UpKeep is a B2B SaaS platform for maintenance and asset management. The product had already found customers when I joined; the engineering problems were about keeping it fast, reliable, and extensible as usage and data grew.My work spanned application development, testing infrastructure, observability, event-driven architecture, search, and database performance — the parts of the system that everything else depended on.
Visit siteHow I contributed
Challenges
- Reporting across MongoDB data
- Reporting workloads needed relational queries across data that lived in MongoDB. We used MoSQL ETL to copy the relevant data into PostgreSQL, where the reporting queries could use relational joins and aggregation. This was part of the database performance work that brought core query times from around 30 seconds down to 10–100ms.
- Tightly coupled application modules
- Features like bulk operations, webhooks, and search all needed to react to the same domain changes, but there was no shared mechanism for propagating them.
- Release risk
- Production releases caused downtime, and there was no environment where a change could be exercised with confidence before it shipped.
- Legacy infrastructure cost and drift
- API servers ran on long-lived legacy machines whose configuration had drifted from anything reproducible, and which cost more than the workload required.
Role & ownership
- OwnedEvent-driven architecture built on Apache Kafka
- OwnedDatabase performance work across PostgreSQL and MongoDB
- OwnedContainerization and migration of API servers off legacy machines
- LedTesting environments and the release process around them
- LedInfrastructure monitoring and scaling
- Contributed toElasticsearch-backed search and suggestions
- Contributed toProduct feature work across the Django and React stack
How it worked
- ClientsAPI servers— requests
- API serversKafka— domain events
- KafkaAsync workers / search indexers— consumers
- WorkersElasticsearch & webhooks— projections, delivery
The API stayed the single writer to the transactional stores. Anything that needed to react to a change — a search index update, a webhook delivery, a bulk operation — consumed a Kafka event instead of being called inline from request handlers.
That separation is what made the rest tractable: read paths could be optimized independently of write paths, Elasticsearch became a projection rather than a second source of truth, and slow or failing consumers stopped being able to slow down a user's request.
Key decisions
- 01
Introduce Kafka as an event backbone rather than adding more direct calls
- Why
- Bulk operations, webhooks, and search all needed the same domain changes. Wiring each one into request handlers would have multiplied coupling and latency.
- Trade-off
- Added a stateful piece of infrastructure to operate, and moved parts of the system to eventual consistency.
- Result
- Modules could be added and changed without touching each other's code paths.
- 02
Use Elasticsearch for full-text search and suggestions instead of extending SQL search
- Why
- Fuzzy matching and suggestions were product requirements that relational search handled poorly at the data volumes involved.
- Trade-off
- A second store to keep in sync, which the event backbone had to guarantee.
- 03
Fix query performance before adding reporting features
- Why
- Roughly 30-second queries meant a KPI dashboard would have been unusable no matter how it was built.
- Result
- Query times moved to the 10–100ms range, which stabilized the system and unblocked the KPI Dashboard.
- 04
Containerize the API servers and migrate off legacy machines
- Why
- Reproducible environments were a prerequisite for reliable releases and for scaling on demand.
- Trade-off
- A migration with no user-visible feature payoff to show for it.
- Result
- Around $2.5K/month saved, with faster and more confident releases.
- 05
Build real testing environments early
- Why
- Releases were causing downtime and new engineers had no safe place to learn the system.
- Result
- Fewer production incidents during releases and noticeably faster onboarding.
Impact
~30s → 10–100ms
Core query latency
~$2.5K/mo
Infrastructure cost saved
What changed
- Reporting became feasible: the KPI Dashboard was built on the optimized query paths.
- Bulk operations, webhooks, and fuzzy search shipped as independent consumers rather than as changes to the core request path.
- Releases stopped being events that required downtime.
- Monitoring made capacity and failure visible instead of inferred from customer reports.
Timeline
- 2018
Joined as Lead Full Stack Engineer
- 2018–2019
Testing environments and monitoring
Made releases repeatable and system behaviour observable.
- 2019
Kafka event-driven architecture
Decoupled bulk operations, webhooks, and search from request handling.
- 2019–2020
Elasticsearch for search and suggestions
- 2020
Containerization and migration off legacy machines
Reproducible API deployments; infrastructure cost reduced.
- 2020–2021
Database performance work
Query latency reduced from ~30s to 10–100ms; KPI Dashboard unblocked.
Artifacts
Looking back
The event backbone paid for itself, but it moved complexity rather than removing it — debugging a consumer is harder than debugging a function call, and that cost lands on whoever is on call.
The infrastructure migration was the least visible work and the highest leverage. It was also the hardest to get prioritized, which says something about how this kind of work usually gets scheduled.
Fixing queries first, before building the dashboard on top of them, was the right order. It is tempting to ship the feature and treat performance as a follow-up; that follow-up rarely happens.