Home

When we scale a neural network to billions of parameters, new capabilities emerge. But we still lack a complete explanation of why certain capabilities emerge or how they arise. Well, this is not unique to neural networks! I recently came across a beautiful essay, “More Is Different,” by Philip W. Anderson. Anderson argued that as systems becom...

Recently, I built a set of interactive simulations for teaching math and science concepts to kids and students. However, I have used Claude to generate such simulations. I wondered, can we build such applets using open-weight models? It turns out that we CAN. We do not need very big models with hundreds of billions of parameters. With an agentic...

My 6-year-old daughter asked me about the phases of the moon. I tried explaining it using pen and paper, hand-waving in the air — she got a bit of it. Like every kid, she watches space cartoons and videos on YouTube. Suddenly I thought of building an interactive simulation like in PhET. I searched PhET, and I didn’t find any app for explaining m...

Given modern language models’ ability to execute actions and their vast accumulation of knowledge, it is compelling to ask: What would happen if AI were to become truly conscious? Conventional wisdom fears a “Terminator” scenario. Yet the outcome might be the opposite. A truly conscious AI may not become a conqueror; instead, it could embody th...

I was asked about the significance of saying “Om”. The friend who asked me this question is a scientist. Before responding, I made sure to understand his intention (genuine interest? yes) I responded by asking him something else. “Are you comfortable with the concept of the span of basis vectors that form a vector space from linear algebra?” ...

Let us train a GPT-2 (small,124 million parameters) model from scratch using the Hugging Face library. Instead of using WebText dataset (due to limited compute resources) I preferred to use the book corpus dataset that contains 74 Million samples (far lower than today’s standard). The book corpus dataset was used to train GPT-1. So, there won’t ...

If you are a researcher or someone who builds/tweaks the deep learning models regularly using the Pytorch framework or any other high-level frameworks that are built on top of Pytorch such as Huggingface, then it is extremely important to understand Pytorch’s nn.Modules. This is because your model could run without displaying any symptoms even i...

Training large language models from scratch is the job of tech giants. Often, the pre-trained proprietary models are adapted to downstream tasks using instruction fine-tuning. However, doing full fine-tuning of model parameters increases the model performance. Of course, full fine- tuning of large models with Billions of parameters requires a go...