The tryst with understanding.

Linear Algebra

Trying to wrap your head around the Matrix as not more that what you solve was tougher than expected. The simple insights "Matrix as linear transformation" and "Linear transformation as matrix" sound trivial and almost explanatory no matter how many times you read them, but getting a stable base around what is when, based on context, is harder to intuit in reality.

Here's the clearest inkling of the understanding I took that made me feel comfortable with the mechanics.

I saw a lecture showing choosing unit vectors as the basis of the transformation, you can see that the area covered by the transformation is actually equal to the determinant of the original matrix. And this question started bugging me to find the proof of how. So I started plotting and solving, the simplest version led me to take an example of a matrix with its non leading diagnols to be 0. Quite trivial, and when I tried to find the area of the resultant matrix, the factorization didn't lead to the obvious det(A) = ad - bd insight. I added realistic scalars to those positions and quickly realised the non - leading diagnols control the shearing/rotation of the axes and the leading diagnols control the scaling factor. And once it was obvious that the most obvious and most general case would yield a parallelogram, I went about trying to solve the equation. Turns out, the lack of practice with Maths made me insufficient to fully solve the equation, with all the tiny bits of the paralleogram spanning multiple portions of the area in question, so I looked up a proof and bam, there it was, in flesh, solved for 2d case and 3d case. And it started to click that the matrix itself can be understood as a transformation itself, taking in a vector as input and moving it into a new coordinate system.

3blue1brown's video on the visual intuition behind the transformation further sealed the deal to understand the mechanical sense of the transformations. But nevertheless, the intuition is still lacking and will solidify only with actual problems I solve in the coming days to understand in what context, what exactly should you do with the matrix. The grounding is the learning.

Coding the forward pass manually

Off late, I have become too reliant of Claude to write code for me. For concepts where coding is actually the blocker, like create server side apps, clis, or frontends or anything in Typescript, its a blesssing as I can offload the slow portion of writing code to Claude. What I didn't realise is that this led to a friction in my mind when it comes to writing code altogether. Take for example, I have run considerable number of experiments on transformers, diffusers and other architectures over the last few months and the amount of understanding of the architecture was almost none. A major part of it was due to the lack of the theroritical lacking in the machine learning theory itself, but a sizeable chunk was also due to me ignoring to write the code. As someone with ADHD, it was hard for me to push my brain through that feeling once the dopamine receptors in my brain started treating it as repetitive and boring.

Cut to last night when I had planned to start dissecting the internals of a model myself, and every single neuron in my brain was fighting to type claude in the terminal and command it to dissect the code for me. I fought the urge and started writing the code myself, asking Claude instead to not help me with the code, but with specific questions about what I wanted to achieve. Starting with a simple print(model), I went on to write the entire forward pass of a trained model and evaluate the shapes. It took me good 4-5 hours to do the whole exercise. But the consolidation of the archtiecture from theory to actually seeing it practice was personally a massive bridge crossed. I had to use my cognition to cycle through the architecture, look at the internals of the transformers package, find the llama model's concrete implementation and by the end, when comparing the logits generated by the model.generate method and my implementation resulted in a np.allcose match of 0, it felt like a triumph.

Don't get me wrong. Its the step 0.001 of where I want to reach, but this step felt the hardest because it didn't need me to overcome complexity, it needed me to overcome me.