write hard things
I’ve worked on hard problems: implemented AI papers, optimized backend services, chased networking issues in production, databases. Some for work, some for fun.
Building hard things gave me understanding and it gave me skills. Everyone talks about the understanding. Almost nobody talks about the skills. I did not know most of them existed until I built something that needed them.
Understanding:
What I cannot create I don’t understand! - Richard Feynman
Skills:
Know how to solve every problem that has been solved. - Richard Feynman
Write hard things. Build a VM from scratch. Write a compiler. Implement a database. I have not found anywhere else that teaches the same things.
Easy advice
Most programming advice is written for easy problems, and every time I took it to a hard one it fell apart.
I tried “keep functions small” on a concurrent state machine. I tried “use design patterns” while optimizing for cache locality. I tried “write tests first” before I knew how to solve the problem at all. None of it worked.
Unit testing did not get me far on distributed systems or databases either. What did: formal methods, deterministic simulation testing, OS-level fault injection, and the kind of deep analysis that actually validates correctness.
The “clean code” abstractions that help on simple problems become the bottleneck in systems programming. A memory allocator isn’t clean. It is pointer arithmetic and bit manipulation and low-level operations, and it can still be readable: the code shows what it does, why, and which invariants it holds.
Simplifying a paper is hard. What has worked for me: implement it first as I understand it, write the tests and the benchmarks, compare it against the alternatives, prove it. Iterate on the invariants and properties until you have squeezed them down to a local minimum. Then simplify: remove state, improve efficiency. Get that far and you have a new result. Publish it.
There is more of this around than I expected. Workflows, manual memory management, concurrency runtimes, network protocols. Public APIs for tools other people depend on. Debugging and performance tracing.
Hard advice
I could not understand any of it without implementing it. I could not simplify anything before making it work. I could not test until I knew what correctness meant there and how to check it reliably. I profiled and debugged a lot. I read assembly, targeted a second architecture, and opened a maths book more than once.
- code: algorithms, CodeForces, competitive programming, Project Euler
- study: databases, distributed systems, operating systems
- hack: hack PyTorch, React, Kubernetes, Postgres, Kafka
- write: start a blog, share guides, write down what you learned, help your teammates
Write hard things.