Artificial Intelligence

One shared activation, five numbers, learned on ImageNet-1K — then frozen and dropped into a 25x smaller Vision Transformer, where it still beats ten fixed activations.

One learnable nonlinearity, shared by every layer, tested head-to-head against a standard fixed activation on real ImageNet-1K.

Every network you’ve trained almost certainly leans on the same handful of activation functions — ReLU, GELU, SiLU, tanh. They differ in shape, but they share one quiet assumption: the nonlinearity itself is fixed. You pick one function, wide enough to cover the whole range of inputs a neuron wil...

When we scale a neural network to billions of parameters, new capabilities emerge. But we still lack a complete explanation of why certain capabilities emerge or how they arise. Well, this is not unique to neural networks! I recently came across a beautiful essay, “More Is Different,” by Philip ...

Recently, I built a set of interactive simulations for teaching math and science concepts to kids and students. However, I have used Claude to generate such simulations. I wondered, can we build such applets using open-weight models? It turns out that we CAN. We do not need very big models with h...

My 6-year-old daughter asked me about the phases of the moon. I tried explaining it using pen and paper, hand-waving in the air — she got a bit of it. Like every kid, she watches space cartoons and videos on YouTube. Suddenly I thought of building an interactive simulation like in PhET. I searche...

Given modern language models’ ability to execute actions and their vast accumulation of knowledge, it is compelling to ask: What would happen if AI were to become truly conscious? Conventional wisdom fears a “Terminator” scenario. Yet the outcome might be the opposite. A truly conscious AI may n...

I was asked about the significance of saying “Om”. The friend who asked me this question is a scientist. Before responding, I made sure to understand his intention (genuine interest? yes) I responded by asking him something else. “Are you comfortable with the concept of the span of basis vecto...

Let us train a GPT-2 (small,124 million parameters) model from scratch using the Hugging Face library. Instead of using WebText dataset (due to limited compute resources) I preferred to use the book corpus dataset that contains 74 Million samples (far lower than today’s standard). The book corpus...

If you are a researcher or someone who builds/tweaks the deep learning models regularly using the Pytorch framework or any other high-level frameworks that are built on top of Pytorch such as Huggingface, then it is extremely important to understand Pytorch’s nn.Modules. This is because your mode...

Training large language models from scratch is the job of tech giants. Often, the pre-trained proprietary models are adapted to downstream tasks using instruction fine-tuning. However, doing full fine-tuning of model parameters increases the model performance. Of course, full fine- tuning of larg...

For the past three months, I have been quite busy building course materials (lecture slides, graded assignments and coding assignments) for the first offering of the course Introduction to Large Language Models by Prof.Mitesh Khapra. It has been challenging work as we have committed to offer the ...

Introduction One question that bothers our mind when we read positional encoding is whether it helps the model or not. It is observed that adding position embeddings only helps marginally in a convolutional neural network as CNN uses relative position embedding (implicitly). It is not the case f...

Data is the fuel for any machine learning model despite the type of learning algorithm (Gradient-based or tree-based) being used. To train and test the model’s generalization capacity, typically, we divide the available samples into three sets: Training, Validation and Test. The typical requireme...

GPT (Generative Pre-trained Transformer) Pre-training Dataset: Book corpus (0.8 Billion words) Unsupervised objective: CLM (Autoregressive) Tokenizer: Byte Pair Encoding (BPE) Vocab size: 40K Architecture: Decoder only (12 Layers) Activation: GELU Attention: Dense FFN: D...

Motivation Usually, in traditional machine learning, we use numerous approaches (model selection) like $K-$fold cross-validation and grid search to find the best model that generalizes well in the real world. However, when it comes to deep learning, it is quite challenging due to compute-cost con...

Introduction I guess you end up being here after coming across the term “constrained optimization” or “Lagrangian” and wanted to understand what “Lagrange multiplier is?”. Well, in this post, I help you understand the foundation of it with interactive plots (you can find plenty of mathematical re...

Introduction The concept of estimation of an unknown quantity from the given observations has been a fascinating area of study for many centuries. However, it remains elusive for many beginners. Let’s start with a concept that we are already familiar with. Here is a sequence $x_1=[1,3,5,7,9,11, \...

Motivation Let’s start with a simple question. How do you represent a number on a real line?. That’s straightforward. How do you represent a 2D point $\begin{bmatrix}x \ y \end{bmatrix}$ in a coordinate system?. Well, we use two real lines that are orthogonal to each other. Move $x$ unit on the a...

Introduction When we have a sequence of elements, we use an index to locate a particular element in the sequence. The sequence of elements can be arranged into 2D,3D or n-dimensional arrays. Therefore, it is extremely important to make ourselves comfortable with using indexes to manipulate such a...

Imagine now that we are just beggining to learn to ride a bicycle. Initially, we need to put a lot of effort into learning to balance the cycle. Then gradually we learn to balance and eventually take a full control over the ride. Suppose that we also wish to learn to ride a simple electric bike. ...

Introduction The Softmax function is one of the most commonly used activation functions at the output layer of neural networks (be it CNN, RNN or Transformers). In fact, Large Language Models(LLMs) based on transformer architecture (like ChatGPT) use softmax in the output layer for many NLP(Natu...

Introduction Masked attention is typically used in the Decoder part of (vanilla) transformer architecture to prevent the model from looking at future tokens. This left me with the impression that the models trained using Masked Language Modelling (MLM) objective use Masked Attention. However, ...

Introduction If you are training a deep learning model or fine-tuning LLMs (Large Language Models), then at some point in time you need to connect with a remote machine that has required amount of computing power (about 80GB or 320 GB of GPU Memory). Data scientists often use Jupyter Notebooks f...

Theory is not enough, you must apply. The objective of this workshop is to give you hands on experience in building models using the pytorch's core component called Tensor . Beleive me, everything you are gonna build is simply stacking or connecting copies of this single core component!. It make...

Gradient as a Guide : A Simple Game   The Backpropagation algorithm is the powerhouse of all Deep Learning models. It is one of the methods of efficiently calculating Gradient for billions of parameters. Therefore, it is vi...

Jupyter notebook has always been a de-facto choice when you teach the Python programming language or while developing prototype machine learning models or doing exploratory data analysis. The reasons are manifold. The most important reason is due to its ability to interleave the rich set of...

Motivation Are you wondering about the peculiar use of a sinusoidal function to encode the positional information in Transformer architecture? Are you asking why not just use simple one-hot encoding or something similar to encode positions?. Welcome, this article is for you. Perhaps you are here...

Initially, I thought of writing an article on transformer architecture. However, I realized that it would go beyond some 90 pages if I cover every detail of it. So I thought of putting the presentation (which is almost self-reliant) prepared by me for the deep learning course taught at IITM by P...

I struggle a lot whenever I learn new mathematical concepts and always sought for some sort of visualization to ease the learning process. I hope I am not alone. We are all comfortable at learning something by looking at the pictures from our childhood onwards. Our brain is wired like that. Howev...

3.RandomProcess A Random process could be thought of as a collection of indexed random variables (i.e., a vector) that evolves in time . Each RV (element) in a collection (vector) follows a specific distribution. If all RVs in a collection have the ...

layout: page keywords: “Estimation, Jupyter notebook” title: “Linear Estimator for a Random Process” date: 2021-05-25 tags: Probability Mathematics key: “LERP-2105” comment: true — 2.CovarianceAndEstimator Oftentimes, statistical averages such a...

1.Random Variables In [1]: import numpy as np from matplotlib import pyplot as plt from matplotlib import transforms as trans from scipy import special plt.style.use('ggplot') In [2]: %matplotlib inline ...