As someone who spends their days thinking about machine learning and their free time pondering life with friends, I can't help but notice the parallels between gradient descent and how we actually navigate uncertainty.
I've often been at a point where the next step was not obvious. Where to study, what to study, where to work, whom to work for... and there will be many more: whom to build a life with, where to live, how to raise children, when to retire, what to do in retirement. There are also smaller daily inflection points: what hobbies to pick up or put down, where to direct effort. The underlying question is always: what should I do next?
I've found that this insecurity is not unique. People's fears are more common than they think, and sharing them is more valuable than they think. When I tell friends about my decision paralysis, the constant re-evaluation, the fear of growing too comfortable or falling for sunk cost fallacy, many recognize and share it. We're all lost hikers at the end of the day.
The Hiker in the Dark
I love Tariq Rashid's introduction to neural networks.1 In Make Your Own Neural Network, he describes gradient descent with a simple analogy: you're a hiker stuck on a mountain at night with only a torch. You can't see the whole landscape. You can only see the ground near your feet. The best you can do is estimate which direction looks like down, take a small step, then look again.
You might get trapped in a local valley that isn't the true bottom. You might have to retrace your steps and start over. But there was no way to know that when you began. At any given moment, you can only take the step that makes sense given what you can see. You can only justify decisions with the information you have right now. It's okay to make mistakes. You couldn't have done better.
When the Rails Disappear
For most of our lives, there's always been a next thing. Preschool, kindergarten, elementary, middle, high school. Then college, with some worry over where or what to study, but still, freshman to sophomore to junior to senior. When we leave, the rails vanish. We don't know what city to work in, who to work for, or what type of job will suit us. We've been dropped into adulthood with no instructions.
When I first moved out at 20, I called my parents and cried that my childhood was over, that I wouldn't casually eat dinner with them at the kitchen table anymore.2 The things I'd taken for granted were gone. It's easy to look to others for answers, but they're stuck traversing their own landscapes too. Filling your time to avoid thinking about it doesn't help. Asking others what you should do doesn't always give you the answers.
What does help is first sitting with that feeling of discomfort, and getting used to it so your decisions aren't made out of fear, uncertainty, and doubt. Then, trying to figure out roughly what you want. What are you optimizing for? What does "going down the mountain" mean for you? Are you chasing prestige or a high-paying career? Looking for friendship, relationships, and community? Focused on personal growth or just having fun? You can change your mind at any time. It's not bad to pick something you thought you wanted and then learn. Even wrong steps give you information about the gradient.
The Way Forward
In optimization problems, we rarely have a bird's-eye view of the entire loss landscape. We only know if we are doing better or worse than before. Life is the same. Just like we can't see the whole function, we can't predict the full consequences of our choices. The temptation is to wait until we have more information, to map everything out before moving, but that map will never come. The landscape is too complex and constantly shifting under our feet. The only way forward is to take steps and notice whether things feel like they are improving.
We may long for a bird's-eye view, but accepting that you are the hiker with the torch in the dark, that you will never see the whole mountain, is freeing. It removes the pressure to have the perfect plan. Your job is not to be right. It is to pay attention and take a step.
We're all lost hikers. The best we can do is shine the torch, look for down, and move.