Artifact</th>	Changes because…</th></tr></thead>
Code</strong></td>	the usual reasons: refactors, bug fixes, new features</td></tr>
Data</strong></td>	new data arrives, distributions drift, labels get corrected</td></tr>
Model</strong></td>	it is retrained, fine-tuned, or replaced</td></tr> </tbody></table> Track only the code and you will still fail to reproduce a model, because the data and the trained model moved underneath you. MLOps is the practice of bringing software-engineering discipline (automation, reproducibility, monitoring) to all three. The one-line definition worth memorising:</p> MLOps = DevOps principles applied to machine-learning systems.</strong></p> </blockquote> To get there, we first need the software-engineering half of that sentence.</p> The software development cycle</h2> Classic software development moves through repeating stages:</p> Development</strong>: planning and coding.</li> Integration</strong>: quick tests and a build.</li> Testing</strong>: deeper tests, validation, and a release.</li> Delivery</strong>: final packaging and deployment to a running server.</li> Monitoring</strong>: collecting data, watching each component, gathering user feedback.</li> </ol> Run those once, by hand, and you have traditional software delivery: slow, batched, and error-prone at the hand-offs.</p> The stages of the software development cycle.</figcaption> </figure> Automating it: DevOps and CI/CD</h2> DevOps is what you get when you automate that cycle</strong> so every change flows through it continuously instead of in big manual batches:</p> Continuous Integration (CI)</strong>: every commit is automatically built and tested.</li> Continuous Delivery (CD)</strong>: passing builds are automatically packaged and deployed.</li> Continuous Monitoring</strong>: the running system is watched for errors and feedback, which flows back into planning.</li> </ul> CI/CD automates the loop: every change is built, tested, and shipped continuously. Credit: resources.github.com/ci-cd</figcaption> </figure> DevOps is usually described as resting on three legs, often drawn as a trident:</p> People</strong>: a culture where development and operations share ownership.</li> Process</strong>: the automated CI/CD pipeline itself.</li> Tools</strong>: the machinery that runs it (GitHub Actions, GitLab CI, containers, and so on).</li> </ul> The tools get the attention, but the culture and the process are what make the automation stick.</p> DevOps rests on three legs: people, process, and tools. Credit: aws.amazon.com/devops</figcaption> </figure> The machine-learning lifecycle</h2> Machine learning adds its own stages on top of the software cycle. A typical ML lifecycle runs:</p> Data extraction</strong>: fetch the data.</li> Data analysis</strong>: understand its nature and quirks.</li> Data preparation</strong>: clean it, engineer features, split into train/validation/test.</li> Model training</strong>: fit the model, tune hyperparameters.</li> Model evaluation</strong>: measure quality on held-out data.</li> Model validation</strong>: confirm it beats a baseline and is fit to deploy.</li> Model serving</strong>: package and deploy it to make predictions.</li> Model monitoring</strong>: watch performance and decide when to retrain.</li> </ol> The machine-learning cycle. Credit: ml-ops.org</figcaption> </figure> The same lifecycle, end to end: data stages feed model stages, and monitoring loops back to the start.</figcaption> </figure> The loop at the end is the important part: monitoring feeds back into data and training. An ML system in production is a living thing, not a shipped artifact.</p> MLOps = DevOps for ML</h2> Now overlay the DevOps loop onto the ML lifecycle. Each "continuous" practice gains an ML twist:</p> MLOps = DevOps principles applied to ML systems. Credit: ml-ops.org</figcaption> </figure> Continuous Integration</strong> is no longer only about testing code: it now also tests and validates data, schemas, and models</strong>.</li> Continuous Delivery</strong> ships not a single package but a whole pipeline</strong> that can deploy a model-serving service.</li> Continuous Training (CT)</strong> is unique to ML: the system can automatically retrain</strong> and redeploy as new data arrives.</li> Continuous Monitoring</strong> tracks model decay</strong> and can trigger retraining when quality drops.</li> </ul> CT and model/data monitoring are what make MLOps its own discipline rather than "DevOps with notebooks".</p> What an MLOps workflow looks like in practice</h2> Put concretely, a working MLOps setup wires together a handful of pieces:</p> code versioning</strong> (git) and data versioning</strong> (DVC): the two moving inputs, tracked like any other asset;</li> experimentation</strong> in notebooks (Jupyter), feeding an experiment-tracking</strong> server (MLflow) that records every run's parameters and metrics;</li> artifact tracking</strong> (also MLflow): every file</em> a run produces (the trained model, a confusion-matrix plot, a preprocessing pipeline) gets stored and linked to the run that made it;</li> a model registry</strong> (MLflow) that versions the best models and stages them (staging → production);</li> pipeline orchestration</strong> (Airflow) that wires the above into a scheduled, repeatable flow;</li> model serving</strong> (BentoML) that wraps a registered model in an API; and</li> model monitoring</strong> (Prometheus/Grafana) that watches the served model and closes the loop back to data.</li> </ul> A concrete MLOps workflow wiring the pieces together. Credit: mymlops.com</figcaption> </figure> Experiment tracking and artifact tracking are easy to conflate; MLflow happens to do both, which is why it appears three times in the diagram. The distinction matters: experiment tracking</strong> logs the numbers</em> (`mlflow.log_metric("accuracy", acc)</code>), while artifact tracking</strong> stores the files</em> a run produced (mlflow.log_artifact("confusion_matrix.png")</code>). Same tool, two different jobs. You will use both in the hands-on below.</p>` `You do not build all of this on day one. The single highest-leverage habit, the one that pays off immediately even in a solo project, is experiment tracking</strong>. That is where we will get hands-on, before bridging to model serving at the end.</p>` MLflow: the tool we will use</h2> MLflow</strong></a> is the open-source standard for experiment tracking. The mental model: an ML experiment is environment + data + code</em>, and MLflow is the logbook that records, for each run, its hyperparameters + results + plots</strong> so you can compare runs later in a web UI. It is framework-agnostic (scikit-learn, PyTorch, TensorFlow, XGBoost…) and works for any</em> computational experiment, not just ML.</p> MLflow has four components:</p> MLflow's components. Credit: mlflow.org</figcaption> </figure> How the pieces fit together when you track an experiment. Credit: mlflow.org</figcaption> </figure> Tracking</strong>: log parameters, metrics, and artifacts per run (our focus).</li> Projects</strong>: package code so a run is reproducible.</li> Models</strong>: a standard packaging format so a trained model can be served anywhere.</li> Model Registry</strong>: version models and move them through stages (staging → production).</li> </ul> The MLflow tracking API</h2> Tracking comes down to a few calls. First, name an experiment to group related runs, then wrap each run in a context manager:</p> import</span> mlflow</span></span> </span> mlflow</span>.</span>set_experiment</span>(</span>"</span>intro-to-mlops</span>"</span>)</span></span> </span> with</span> mlflow</span>.</span>start_run</span>(</span>run_name</span>=</span>"</span>my-first-run</span>"</span>)</span>:</span></span> mlflow</span>.</span>log_param</span>(</span>"</span>learning_rate</span>"</span>,</span> 0.01</span>)</span> #</span> one hyperparameter</span></span> mlflow</span>.</span>log_params</span>(</span>{</span>"</span>epochs</span>"</span>:</span> 20</span>,</span> "</span>batch</span>"</span>:</span> 32</span>}</span>)</span> #</span> several at once</span></span> mlflow</span>.</span>log_metric</span>(</span>"</span>accuracy</span>"</span>,</span> 0.92</span>)</span> #</span> a result</span></span> mlflow</span>.</span>sklearn</span>.</span>log_model</span>(</span>model</span>,</span> name</span>=</span>"</span>model</span>"</span>)</span> #</span> the model itself</span></span></code></pre> The three verbs you will use constantly:</p> log_param</code> / log_params</code>: key-value settings (hyperparameters, config).</li> log_metric</code>: numeric results (loss, accuracy); these become plots in the UI so you can compare runs.</li> log_model</code>: the trained model, in MLflow's portable format.</li> </ul> Too lazy to log by hand?</strong> For supported libraries, a single autolog()</code> call before fit()</code> captures parameters, metrics, and the model automatically:</p> mlflow</span>.</span>sklearn</span>.</span>autolog</span>(</span>)</span> #</span> or mlflow.tensorflow.autolog(), mlflow.pytorch.autolog()</span></span> model</span>.</span>fit</span>(</span>X_train</span>,</span> y_train</span>)</span> #</span> params, metrics, and model are logged for you</span></span></code></pre> Manual logging gives you the most control; autolog</code> gives you most of the value for one line. And the feature people quietly love most: logging plots as artifacts</strong>, so a figure is stored with the run and viewable in the UI:</p> plt</span>.</span>savefig</span>(</span>"</span>loss.png</span>"</span>)</span></span> mlflow</span>.</span>log_artifact</span>(</span>"</span>loss.png</span>"</span>)</span></span></code></pre>Where MLflow stores runs: three setup options</h2> Before logging anything, MLflow needs to know where</em> runs go. There are three options, in increasing order of scale:</p> MLflow tracking setup. We use the local-database option below. Credit: mlflow.org</figcaption> </figure> Option 1, local filesystem</strong> (./mlruns</code>): the simplest default.</li> Option 2, local database</strong> (sqlite:///mlflow.db</code>): still on your laptop, but the backend MLflow now recommends. This is what we use below.</strong></li> Option 3, remote server</strong> (http://server:5000</code>): a shared tracking server for a team.</li> </ul> Hands-on: track your first experiment</h2> Everything below is self-contained and runs on a laptop in under ten minutes. We will train two small classifiers, log them to MLflow, and compare them in the UI.</p> Prerequisites:</strong> Python 3.9+ and pip</code>. No GPU, no cloud account, no prior MLflow setup. Five steps: install → write the script → run it → run it again → open the UI.</p> Want to just clone and run?</strong> The full source (script and</strong> notebook) lives in the companion repo: github.com/JienWeng/mlops-tutorial</strong></a>. git clone</code> it, pip install -r requirements.txt</code>, then python train.py</code> (or open notebook.ipynb</code>). The steps below explain what that code does.</p> </blockquote> 1. Install</h3> python</span> -</span>m</span> pip</span> install</span> mlflow</span> scikit-learn</span> matplotlib</span></span></code></pre>2. Choose a store (Option 2: local SQLite)</h3> We use a local SQLite database, Option 2 above. It keeps everything on your laptop with no server to run.</p> Heads-up (MLflow 3.x):</strong> the plain-folder store (./mlruns</code> on its own) is now in maintenance mode and will raise an error unless you opt in. A SQLite URI like sqlite:///mlflow.db</code> is the friction-free local setup, so that is what we use. (On MLflow 2.x this same code still works.)</p> </blockquote> We set the store directly in the script, so there is nothing to configure separately.</p> 3. The experiment script</h3> Save this as train.py</code>. It trains a logistic-regression classifier on the classic breast-cancer dataset, then logs the parameters, the metrics, the trained model, and a confusion-matrix plot.</p> #</span> train.py</span></span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span> import</span> mlflow</span></span> import</span> mlflow</span>.</span>sklearn</span></span> from</span> sklearn</span>.</span>datasets</span> import</span> load_breast_cancer</span></span> from</span> sklearn</span>.</span>linear_model</span> import</span> LogisticRegression</span></span> from</span> sklearn</span>.</span>metrics</span> import</span> ConfusionMatrixDisplay</span>,</span> accuracy_score</span>,</span> f1_score</span></span> from</span> sklearn</span>.</span>model_selection</span> import</span> train_test_split</span></span> </span> #</span> Where to store runs: a local SQLite db (metadata) + ./mlruns (artifacts)</span></span> mlflow</span>.</span>set_tracking_uri</span>(</span>"</span>sqlite:///mlflow.db</span>"</span>)</span></span> #</span> Group related runs under a named experiment</span></span> mlflow</span>.</span>set_experiment</span>(</span>"</span>intro-to-mlops</span>"</span>)</span></span> </span> #</span> --- data ---</span></span> X</span>,</span> y</span> =</span> load_breast_cancer</span>(</span>return_X_y</span>=</span>True</span>)</span></span> X_train</span>,</span> X_test</span>,</span> y_train</span>,</span> y_test</span> =</span> train_test_split</span>(</span></span> X</span>,</span> y</span>,</span> test_size</span>=</span>0.2</span>,</span> random_state</span>=</span>42</span></span> )</span></span> </span> #</span> A hyperparameter we want to track and vary later</span></span> C</span> =</span> 1.0</span> #</span> inverse regularisation strength</span></span> </span> with</span> mlflow</span>.</span>start_run</span>(</span>run_name</span>=</span>f</span>"</span>logreg-C=</span>{</span>C</span>}</span>"</span>)</span>:</span></span> #</span> --- train ---</span></span> model</span> =</span> LogisticRegression</span>(</span>C</span>=</span>C</span>,</span> max_iter</span>=</span>10_000</span>)</span></span> model</span>.</span>fit</span>(</span>X_train</span>,</span> y_train</span>)</span></span> </span> #</span> --- evaluate ---</span></span> preds</span> =</span> model</span>.</span>predict</span>(</span>X_test</span>)</span></span> acc</span> =</span> accuracy_score</span>(</span>y_test</span>,</span> preds</span>)</span></span> f1</span> =</span> f1_score</span>(</span>y_test</span>,</span> preds</span>)</span></span> </span> #</span> --- log: parameters, metrics, the model itself ---</span></span> mlflow</span>.</span>log_param</span>(</span>"</span>model</span>"</span>,</span> "</span>LogisticRegression</span>"</span>)</span></span> mlflow</span>.</span>log_param</span>(</span>"</span>C</span>"</span>,</span> C</span>)</span></span> mlflow</span>.</span>log_metric</span>(</span>"</span>accuracy</span>"</span>,</span> acc</span>)</span></span> mlflow</span>.</span>log_metric</span>(</span>"</span>f1</span>"</span>,</span> f1</span>)</span></span> mlflow</span>.</span>sklearn</span>.</span>log_model</span>(</span>model</span>,</span> name</span>=</span>"</span>model</span>"</span>)</span></span> </span> #</span> --- log: a plot as an artifact ---</span></span> ConfusionMatrixDisplay</span>.</span>from_predictions</span>(</span>y_test</span>,</span> preds</span>)</span></span> plt</span>.</span>title</span>(</span>f</span>"</span>Confusion matrix (C=</span>{</span>C</span>}</span>)</span>"</span>)</span></span> plt</span>.</span>savefig</span>(</span>"</span>confusion_matrix.png</span>"</span>,</span> bbox_inches</span>=</span>"</span>tight</span>"</span>)</span></span> mlflow</span>.</span>log_artifact</span>(</span>"</span>confusion_matrix.png</span>"</span>)</span></span> </span> print</span>(</span>f</span>"</span>Logged run: accuracy=</span>{</span>acc</span>:.4f</span>}</span>, f1=</span>{</span>f1</span>:.4f</span>}</span>"</span>)</span></span></code></pre> Run it:</p> python</span> train.py</span></span></code></pre> You just produced your first tracked run. The parameters, metrics, the serialised model, and the plot are all saved.</p> 4. Make a second run to compare against</h3> Change one line, C = 0.01</code>, and run python train.py</code> again. (Stronger regularisation; the score should move.) Now you have two runs to compare</strong>, which is the whole point of tracking.</p> 5. Inspect and compare in the UI</h3> From the same folder, start the UI, pointing it at the same SQLite store:</p> mlflow</span> ui</span> -</span>-backend-store-uri</span> sqlite:///mlflow.db</span></span></code></pre> Open http://127.0.0.1:5000</a>. You will see the intro-to-mlops</code> experiment with both runs. It looks like this:</p> The MLflow UI: every run with its parameters and metrics, ready to sort and compare. Credit: mlflow.org</figcaption> </figure> From here you can:</p> sort and filter runs by accuracy</code> or f1</code>,</li> tick two runs and click Compare</strong> to see parameters and metrics side by side,</li> open a run to view its confusion-matrix plot</strong> and download the saved model.</li> </ul> That comparison view answers the question MLOps exists to answer: which settings gave the best score, and exactly how were they produced? You now have a reproducible record of it.</p> 6. Serve it: MLflow → BentoML (minimal)</h3> Tracking answers "what's the best run?" Serving answers "how do users call it?" This is the model serving</strong> box from the workflow diagram above, and BentoML</strong></a> is the tool. The bridge is one call: load the model MLflow already tracked and stored, hand it to a Bento service.</p> python</span> -</span>m</span> pip</span> install</span> bentoml</span></span></code></pre> Save this as service.py</code>, next to mlflow.db</code> and mlruns/</code> (also in the companion repo, alongside train.py</code>):</p> #</span> service.py</span></span> import</span> bentoml</span></span> import</span> mlflow</span></span> </span> mlflow</span>.</span>set_tracking_uri</span>(</span>"</span>sqlite:///mlflow.db</span>"</span>)</span></span> </span> #</span> Grab the most recent run of the experiment, no run ID to copy by hand</span></span> experiment</span> =</span> mlflow</span>.</span>get_experiment_by_name</span>(</span>"</span>intro-to-mlops</span>"</span>)</span></span> latest_run</span> =</span> mlflow</span>.</span>search_runs</span>(</span></span> experiment</span>.</span>experiment_id</span>,</span> order_by</span>=</span>[</span>"</span>start_time DESC</span>"</span>]</span>,</span> max_results</span>=</span>1</span></span> )</span>.</span>iloc</span>[</span>0</span>]</span></span> model</span> =</span> mlflow</span>.</span>sklearn</span>.</span>load_model</span>(</span>f</span>"</span>runs:/</span>{</span>latest_run</span>.</span>run_id</span>}</span>/model</span>"</span>)</span></span> </span> @</span>bentoml</span>.</span>service</span></span> class</span> MLopsService</span>:</span></span> @</span>bentoml</span>.</span>api</span></span> def</span> predict</span>(</span>self</span>,</span> features</span>:</span> list</span>[</span>list</span>[</span>float</span>]</span>]</span>)</span> -></span> list</span>[</span>int</span>]</span>:</span></span> return</span> model</span>.</span>predict</span>(</span>features</span>)</span>.</span>tolist</span>(</span>)</span></span></code></pre> Run it:</p> bentoml</span> serve</span> service:MLopsService</span></span></code></pre> That starts an HTTP API on http://127.0.0.1:3000</a> with interactive docs. Call it:</p> curl</span> -</span>X</span> POST</span> http://127.0.0.1:3000/predict</span> \</span></span> -</span>H</span> "</span>Content-Type: application/json</span>"</span> \</span></span> -</span>d</span> '</span>{"features": [[14.0, 20.0, 90.0, 600.0, 0.1, 0.1, 0.1, 0.05, 0.2, 0.06, 0.4, 1.0, 3.0, 40.0, 0.006, 0.02, 0.03, 0.01, 0.02, 0.003, 16.0, 25.0, 105.0, 900.0, 0.14, 0.25, 0.3, 0.12, 0.3, 0.08]]}</span>'</span></span></code></pre> That's the whole bridge: MLflow tracks and stores the model; BentoML loads it by URI and serves it.</strong> Everything past this (batching, Docker packaging with bentoml build</code>, autoscaling) is the same idea scaled up.</p> What you just built (and what comes next)</h3> You have implemented the smallest real MLOps loop: parameterise → run → log → compare → serve</strong>. The natural extensions, in rough order of payoff:</p> Commit train.py</code> to git</strong> so each run is tied to a code version.</li> Register the best model</strong> in MLflow's Model Registry to give it a name and version, instead of pasting a run ID.</li> Automate the run in CI</strong> so a push retrains and re-logs.</li> Monitor</strong> the served model's predictions, closing the loop back to data.</li> </ol> That list is exactly the MLOps workflow from earlier. You have now built its first two links.</p> Take-home message</h2> Research, and production work too, should be reproducible</strong> and, where possible, open</strong>. If your work involves machine learning, that means using dedicated tools to make experiments reproducible rather than relying on memory and scattered notebooks.</p> Adopting these habits in your daily workflow pays off quickly: time saved in the long run, engineering skill gained, and trust earned among collaborators. And MLflow in particular is not just for machine learning</strong>: it is a capable logbook for tracking any</em> computational experiment and keeping your results reproducible over time.</p> .small[This tutorial follows and is adapted from the lecture "An introduction to MLOps"</strong> by Alexandre Boucaud</a> (LSST France, Lyon, December 2023), licensed under CC BY-SA 4.0</a>. The figures are taken from the accompanying slides</a>, which are re-hosted under the same license; individual figures credit their original sources (ml-ops.org, mlflow.org, mymlops.com, GitHub, AWS) in their captions. This written tutorial and hands-on are an original adaptation, likewise shared under CC BY-SA 4.0.]</p> Tree-Based Models Explained: From Decision Trees to Random Forest and XGBoost 2026-05-11T00:00:00+00:00 A single decision tree is powerful but unstable. Nearly everything about Random Forests and XGBoost follows from that one sentence, so that is where this note starts, not at the formulas. Forests and boosting are just two different answers to the instability problem.</p> The path is:</p> $$ \text{decision rules} \to \text{decision trees} \to \text{bagging} \to \text{Random Forest} \to \text{boosting} \to \text{XGBoost}. $$</p> Intuition first, but the math is written out in full. I dislike formulas that appear from nowhere.</p> Run it yourself.</strong> The worked example is collected in a runnable notebook: download tree-based-models.ipynb</code></strong></a>. It uses numpy</code>, scikit-learn</code>, and (optionally) xgboost</code>.</p> </blockquote> Why People Reach for Trees, Random Forests, and XGBoost</h2> The motivation usually looks like this:</p> Linear models miss nonlinear structure.</strong> Trees capture nonlinear, rule-like boundaries without feature engineering.</li> Single trees are unstable.</strong> They are accurate but high-variance.</li> Random Forests stabilize trees.</strong> Bagging reduces variance by averaging many noisy trees.</li> Boosting repairs bias.</strong> XGBoost builds trees sequentially to correct systematic mistakes.</li> </ul> So the choice is not "trees vs. XGBoost". It is more like this:</p> Use a single tree</strong> to explain a small, interpretable rule set.</li> Use a Random Forest</strong> when you want strong performance with low tuning risk.</li> Use XGBoost</strong> when you want maximum accuracy and can tune regularization.</li> </ul> Start with the Simplest Useful Model: A Decision Rule</h2> Imagine a classifier that uses a single rule:</p> $$ \text{if } x_1 \le 3.5, \text{ predict class 0, else class 1}. $$</p> This is a decision stump. It is weak, but it is easy to interpret. The point is not accuracy. The point is that rules like this are already nonlinear in the raw feature space.</p> Now chain many rules in sequence, and you get a decision tree.</p> A single decision rule creates a nonlinear boundary from a linear threshold.</figcaption> </figure> Decision Trees as Greedy Partitioning</h2> A decision tree repeatedly splits the feature space into regions. Each split chooses a feature and a threshold. The goal is to make the resulting child nodes more "pure" than the parent node.</p> For classification, the standard impurity measures are entropy</strong> and Gini</strong>.</p> Entropy for a node with class proportions $p_k$ is</p> $$ H(S) = -\sum_{k=1}^{K} p_k \log p_k. $$</p> Gini impurity is</p> $$ G(S) = 1 - \sum_{k=1}^{K} p_k^2. $$</p> Both measures are $0$ when a node is perfectly pure and larger when it is mixed.</p> Information Gain</h3> Suppose a candidate split divides a node $S$ into children $S_L$ and $S_R$. Let $n$ be the total samples in $S$, with $n_L$ and $n_R$ in the children. The information gain</strong> is</p> $$ \text{IG}(S, \text{split}) = H(S) - \frac{n_L}{n} H(S_L) - \frac{n_R}{n} H(S_R). $$</p> Using Gini, replace $H$ with $G$ in the same formula.</p> The tree algorithm is greedy: at each node it chooses the split that maximizes this gain.</p> A Small Example</h3> Assume a node has $10$ samples: $6$ of class 1 and $4$ of class 0. The entropy is</p> $$ H(S) = -\left(\frac{6}{10}\log\frac{6}{10} + \frac{4}{10}\log\frac{4}{10}\right). $$</p> Now imagine a split that creates:</p> Left child: $4$ class 1, $1$ class 0 ($n_L=5$)</li> Right child: $2$ class 1, $3$ class 0 ($n_R=5$)</li> </ul> Compute $H(S_L)$ and $H(S_R)$, and you get a positive information gain. The split is not perfect, but it is better than the parent. That is enough to move the algorithm forward.</p> A split is chosen because it reduces impurity, not because it is globally optimal.</figcaption> </figure> Leaf Predictions</h3> Once a node stops splitting, it becomes a leaf. For classification, the leaf prediction is</p> $$ \hat{p}_k = \frac{\text{count of class } k \text{ in leaf}}{\text{samples in leaf}}, $$</p> and the predicted class is the argmax of $\hat{p}_k$.</p> For regression, the leaf prediction is typically the mean:</p> $$ \hat{y} = \frac{1}{\|S\|} \sum_{i \in S} y_i. $$</p> Why Trees Are Unstable</h3> A tree is a high-variance model. A small change in the dataset can change an early split, which changes every downstream region. Trees can overfit without strong regularization such as max depth, minimum samples per leaf, or pruning.</p> This is where ensembles enter the story.</p> Bagging: Reduce Variance by Averaging</h2> Bagging (bootstrap aggregating) trains many trees on different bootstrap samples of the data and averages their predictions.</p> Let $\hat{f}_1, \dots, \hat{f}_M$ be $M$ trees. The bagged predictor is</p> $$ \hat{f}_{\text{bag}}(x) = \frac{1}{M} \sum^M \hat{f}_m(x). $$</p> If each tree has variance $\sigma^2$ and pairwise correlation $\rho$, then</p> $$ \text{Var}(\hat{f}_{\text{bag}}) = \frac{1}{M}(1-\rho)\sigma^2 + \rho\sigma^2. $$</p> So averaging reduces variance, but correlation limits how much we gain.</p> Random Forest: Bagging + Feature Randomness</h2> Random Forests go one step further. They reduce correlation by forcing each split to consider only a random subset of features.</p> This does two things:</p> It decorrelates the trees, which reduces the $\rho$ term above.</li> It prevents strong predictors from dominating every split.</li> </ol> The forest prediction is still an average (regression) or a majority vote (classification). The math is the same as bagging, but the diversity is higher.</p> Random Forests reduce variance by averaging many decorrelated trees.</figcaption> </figure> Boosting: Reduce Bias by Building Sequentially</h2> Bagging attacks variance. Boosting attacks bias.</p> The idea is to build an additive model that corrects its own mistakes:</p> $$ \hat{y}^{(t)}(x) = \hat{y}^{(t-1)}(x) + f_t(x), $$</p> where each $f_t$ is a small tree and the sequence is trained to reduce a loss function.</p> Boosting is best understood as functional gradient descent</strong>: we are minimizing a loss by moving in function space, one tree at a time.</p> XGBoost: Boosting with a Second-Order Objective</h2> XGBoost formalizes this with an objective that includes regularization. At step $t$:</p> $$ \mathcal{L}^{(t)} = \sum_{i=1}^n l\left(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)\right) + \Omega(f_t), $$</p> where $l$ is the training loss and the regularization term is</p> $$ \Omega(f) = \gamma T + \frac{1}{2}\lambda \sum_{j=1}^{T} w_j^2. $$</p> Here $T$ is the number of leaves and $w_j$ is the score assigned to leaf $j$.</p> Second-Order Approximation</h3> Let</p> $$ g_i = \partial_{\hat{y}} l(y_i, \hat{y}_i^{(t-1)}), $$</p> $$ h_i = \partial_{\hat{y}}^2 l(y_i, \hat{y}_i^{(t-1)}). $$</p> Using a second-order Taylor expansion around $\hat{y}_i^{(t-1)}$, the objective becomes</p> $$ \tilde{\mathcal{L}}^{(t)} = \sum_{i=1}^n \left[g_i f_t(x_i) + \frac{1}{2} h_i f_t(x_i)^2\right] + \Omega(f_t). $$</p> Because a tree predicts a constant weight $w_j$ for all samples in leaf $j$, we can group terms per leaf. Let $I_j$ be the indices in leaf $j$ and define</p> $$ G_j = \sum_{i \in I_j} g_i, \qquad H_j = \sum_{i \in I_j} h_i. $$</p> Then the optimal leaf weight is</p> $$ w_j^* = -\frac{G_j}{H_j + \lambda}, $$</p> and the best value of the objective for that leaf is</p> $$ -\frac{1}{2} \frac{G_j^2}{H_j + \lambda}. $$</p> Split Gain</h3> A split is good if it improves the objective. The gain of splitting a node into left and right children is</p> $$ \text{Gain} = \frac{1}{2}\left(\frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{G^2}{H + \lambda}\right) - \gamma. $$</p> This is the exact reason XGBoost splits where it does. The split is not only about purity. It is about how much the second-order objective improves after regularization.</p> XGBoost chooses splits by maximizing second-order gain with regularization.</figcaption> </figure> Why XGBoost Works So Well</h2> The success of XGBoost is not a mystery. It is the combination of:</p> Sequential correction:</strong> each tree targets the remaining error.</li> Second-order curvature:</strong> Hessians stabilize the optimization.</li> Regularization:</strong> the $\gamma$ and $\lambda$ terms actively penalize complexity.</li> Shrinkage:</strong> a learning rate scales each tree so the model does not overreact.</li> </ul> In practice, these ideas mean the model stays expressive without exploding in variance.</p> Boosting builds an additive model one small tree at a time.</figcaption> </figure> A Few Practical Tradeoffs</h2> Tree ensembles are strong, but their strengths are specific:</p> Interpretability:</strong> single trees are readable, forests and boosted trees are not.</li> Bias vs variance:</strong> Random Forests reduce variance, boosting reduces bias.</li> Overfitting:</strong> deep trees can overfit, especially in boosting without regularization.</li> Data leakage:</strong> trees can memorize target leakage quickly, so features must be clean.</li> Class imbalance:</strong> impurity is dominated by the majority class unless weighted.</li> Compute cost:</strong> XGBoost is fast for trees, but still heavier than linear models.</li> </ul> A Worked Dataset Example (Synthetic Regression)</h2> To make this concrete, we use a small synthetic regression dataset from scikit-learn. It is numeric, quick to run, and avoids any external downloads. The idea is to compare a single tree, a Random Forest, and XGBoost on the same train/test split.</p> A synthetic regression dataset with noise and nonlinear structure.</figcaption> </figure> Below is a compact code snippet. The full notebook</a> is provided separately so you can run it end-to-end.</p> import</span> numpy</span> as</span> np</span></span> from</span> sklearn</span>.</span>datasets</span> import</span> make_regression</span></span> from</span> sklearn</span>.</span>model_selection</span> import</span> train_test_split</span></span> from</span> sklearn</span>.</span>metrics</span> import</span> mean_absolute_error</span>,</span> mean_squared_error</span></span> from</span> sklearn</span>.</span>tree</span> import</span> DecisionTreeRegressor</span></span> from</span> sklearn</span>.</span>ensemble</span> import</span> RandomForestRegressor</span></span> </span> try</span>:</span></span> from</span> xgboost</span> import</span> XGBRegressor</span></span> except</span> ImportError</span>:</span></span> XGBRegressor</span> =</span> None</span></span> </span> X</span>,</span> y</span> =</span> make_regression</span>(</span>n_samples</span>=</span>2000</span>,</span> n_features</span>=</span>12</span>,</span> noise</span>=</span>15.0</span>,</span> random_state</span>=</span>42</span>)</span></span> X_train</span>,</span> X_test</span>,</span> y_train</span>,</span> y_test</span> =</span> train_test_split</span>(</span></span> X</span>,</span> y</span>,</span> test_size</span>=</span>0.2</span>,</span> random_state</span>=</span>42</span></span> )</span></span> </span> tree</span> =</span> DecisionTreeRegressor</span>(</span>max_depth</span>=</span>4</span>,</span> random_state</span>=</span>42</span>)</span></span> rf</span> =</span> RandomForestRegressor</span>(</span>n_estimators</span>=</span>300</span>,</span> random_state</span>=</span>42</span>)</span></span> </span> models</span> =</span> {</span>"</span>tree</span>"</span>:</span> tree</span>,</span> "</span>rf</span>"</span>:</span> rf</span>}</span></span> if</span> XGBRegressor</span> is</span> not</span> None</span>:</span></span> xgb</span> =</span> XGBRegressor</span>(</span></span> n_estimators</span>=</span>400</span>,</span></span> max_depth</span>=</span>4</span>,</span></span> learning_rate</span>=</span>0.05</span>,</span></span> subsample</span>=</span>0.8</span>,</span></span> colsample_bytree</span>=</span>0.8</span>,</span></span> reg_lambda</span>=</span>1.0</span>,</span></span> random_state</span>=</span>42</span>,</span></span> )</span></span> models</span>[</span>"</span>xgb</span>"</span>]</span> =</span> xgb</span></span> </span> for</span> name</span>,</span> model</span> in</span> models</span>.</span>items</span>(</span>)</span>:</span></span> model</span>.</span>fit</span>(</span>X_train</span>,</span> y_train</span>)</span></span> preds</span> =</span> model</span>.</span>predict</span>(</span>X_test</span>)</span></span> rmse</span> =</span> np</span>.</span>sqrt</span>(</span>mean_squared_error</span>(</span>y_test</span>,</span> preds</span>)</span>)</span></span> mae</span> =</span> mean_absolute_error</span>(</span>y_test</span>,</span> preds</span>)</span></span> print</span>(</span>name</span>,</span> "</span>rmse</span>"</span>,</span> round</span>(</span>rmse</span>,</span> 3</span>)</span>,</span> "</span>mae</span>"</span>,</span> round</span>(</span>mae</span>,</span> 3</span>)</span>)</span></span></code></pre> RMSE and MAE comparison across a single tree, Random Forest, and XGBoost.</figcaption> </figure> Feature importance comparison for Random Forest and XGBoost.</figcaption> </figure> What Code to Look At</h2> If you want to read real implementations, start here:</p> sklearn.tree.DecisionTreeClassifier</code> for tree splits, impurity, and pruning logic.</li> sklearn.ensemble.RandomForestClassifier</code> for bagging and feature subsampling.</li> xgboost.XGBClassifier</code> for second-order boosting with regularization.</li> </ul> The Big Picture</h2> We started with simple decision rules. That gave us decision trees.</p> Decision trees are powerful but unstable, so we average them. That gives us bagging, and then Random Forests.</p> Random Forests reduce variance but still leave bias on the table, so we build trees sequentially to correct errors. That gives us boosting, and XGBoost.</p> Trees define nonlinear partitions, bagging stabilizes them, boosting makes them accurate. That is the whole story, and every formula above is one of those three moves written out.</p> At some point I may extend the notebook with hyperparameter tuning and a multi-class example. If you want that sooner, leave a comment.</p> Fast Fourier Transform Explained from the DFT, with a Sound-Wave Example 2026-04-22T00:00:00+00:00 The butterfly diagram is usually the first thing people show you about the FFT. I think it should be the last. The FFT is nothing more than a fast way to compute the Discrete Fourier Transform, so none of it makes sense until the DFT does. And the DFT itself is easiest to see if you start from a sampled sound wave.</p> So this note goes:</p> $$ \text{sound wave} \to \text{samples} \to \text{DFT} \to \text{FFT} \to \text{spectrum}. $$</p> We build up the picture of sound as a sum of sinusoids, get the DFT as its discrete version, and then let the structure of the DFT matrix hand us the Cooley-Tukey FFT almost for free. At the end we run the whole thing on a real sound wave, implemented from scratch in Python.</p> Run it yourself.</strong> Every code block below is collected in a runnable Jupyter notebook: download fft-sound-wave.ipynb</code></strong></a>. It only needs numpy</code> and matplotlib</code> (pip install numpy matplotlib</code>), and runs top to bottom.</p> </blockquote> Start with a Sound Wave</h2> Think of a pure musical tone. When a tuning fork vibrates at $440$ Hz, it displaces air molecules in a nearly sinusoidal way. The pressure at a fixed point in space looks like</p> $$ x(t) = A \cos(2\pi f t + \varphi), $$</p> with frequency $f = 440$ Hz, some amplitude $A$, and some phase $\varphi$.</p> A real sound is almost never a single sinusoid. A piano note at A4 contains the fundamental at $440$ Hz plus overtones at roughly $880$ Hz, $1320$ Hz, and so on, each with its own amplitude. More generally, any reasonable sound can be written as a sum of sinusoids:</p> $$ x(t) = \sum_{k} A_k \cos(2\pi f_k t + \varphi_k). $$</p> This is the Fourier picture. The time-domain view tells you what the waveform looks like at each instant. The frequency-domain view tells you which sinusoids, at which amplitudes and phases, combine to produce that waveform. The two views carry the same information but answer different questions.</p> On a computer, we never see $x(t)$ directly. We see samples. An analogue-to-digital converter records the pressure at a fixed sample rate $f_s$ (for example $44{,}100$ samples per second for CD audio), producing a finite sequence</p> $$ x_n = x(n / f_s), \qquad n = 0, 1, \ldots, N-1. $$</p> The question we really want to answer is: given these $N$ samples, which frequencies are present and how strong are they?</p> That question has a precise answer called the Discrete Fourier Transform.</p> From Fourier Series to the DFT</h2> The continuous Fourier series says that a periodic signal can be written as a sum of complex exponentials:</p> $$ x(t) = \sum_{k=-\infty}^{\infty} c_k , e^{i 2\pi k t / T}, $$</p> where $T$ is the period. The coefficients $c_k$ tell us how much of each frequency $k/T$ is present. Complex exponentials show up here for a reason. Using Euler's formula,</p> $$ e^{i\theta} = \cos\theta + i\sin\theta, $$</p> a single complex exponential carries both a cosine and a sine at the same frequency. That packages amplitude and phase into one object and lets the math stay linear and clean.</p> Now discretize. Suppose we sample $x(t)$ at $N$ evenly spaced points over one period $T$, with spacing $\Delta t = T/N$. The samples are</p> $$ x_n = x(n \Delta t), \qquad n = 0, 1, \ldots, N-1. $$</p> The natural discrete analogue of the Fourier series uses $N$ complex exponentials,</p> $$ e^{i 2\pi k n / N}, \qquad k = 0, 1, \ldots, N-1, $$</p> as a basis for length-$N$ sequences. These discrete exponentials are orthogonal:</p> $$ \sum_{n=0}^{N-1} e^{i 2\pi k n / N} , e^{-i 2\pi \ell n / N} = \begin{cases} N & \text{if } k = \ell, \ 0 & \text{otherwise.} \end{cases} $$</p> Orthogonality is what makes the transform invertible. It also tells us the right formula for the coefficients, by projecting $x_n$ onto each basis vector.</p> This gives the Discrete Fourier Transform. For a sequence $x_0, x_1, \ldots, x_{N-1}$, define</p> $$ X_k = \sum_{n=0}^{N-1} x_n , e^{-i 2\pi k n / N}, \qquad k = 0, 1, \ldots, N-1. $$</p> And the inverse:</p> $$ x_n = \frac{1}{N} \sum_{k=0}^{N-1} X_k , e^{i 2\pi k n / N}. $$</p> Each $X_k$ is a complex number. Its magnitude $\|X_k\|$ tells us how strong the frequency component at bin $k$ is, and its phase $\arg X_k$ tells us where that sinusoid starts.</p> The frequency associated with bin $k$ is</p> $$ f_k = \frac{k}{N} f_s, $$</p> for $k = 0, 1, \ldots, N/2$, after which the bins fold by symmetry for real inputs. The highest frequency we can resolve is $f_s / 2$, the Nyquist frequency. Anything above that aliases down into lower bins, which is why sample rate matters.</p> What the Transform Actually Does: The Winding View</h2> The formula for $X_k$ is easy to write down but easy to stare at without seeing anything. Before we optimize how</em> to compute it, here is a different perspective on what</em> it computes, one that makes the whole transform intuitive.</p> Look again at a single term. The factor</p> $$ e^{-i 2\pi k n / N} $$</p> is a unit vector in the complex plane that rotates clockwise as $n$ advances. Multiplying the signal $x_n$ by it winds the signal around the origin</strong> at a winding rate set by $k$: small $k$ winds slowly, large $k$ winds fast. The sum over $n$ then just adds up all those wound points, and dividing by $N$ gives their center of mass</strong> (their average position).</p> So $X_k / N$ is literally: take the signal, wrap it around a circle at rate $k$, and find the center of mass of the resulting curve.</em></p> Now the key effect. When the winding rate matches</strong> a frequency that is actually present in the signal, the signal's peaks land on the same side of the circle on every turn. The wound points pile up on one side, and their center of mass sits far from the origin: a large $\|X_k\|$. When the winding rate does not match</strong>, the peaks land at different angles each turn, spread evenly around the circle, and cancel. The center of mass collapses to near zero.</p> To see it, here is a simple two-component signal (a $2$ Hz plus a $3$ Hz cosine, chosen at low frequencies purely so the winding is visible) wound at four different rates:</p> Winding a 2 Hz + 3 Hz signal. At 2 Hz and 3 Hz the center of mass (red) pulls away from the origin; at 1 Hz and 2.5 Hz it cancels back to the center.</figcaption> </figure> Sweep the winding rate continuously and record how far the center of mass sits from the origin at each rate, and you trace out the Fourier transform magnitude directly: flat and near zero everywhere except sharp bumps at exactly the frequencies the signal contains.</p> Sweeping the winding frequency reproduces the spectrum: peaks at 2 and 3 Hz, near-zero elsewhere. This curve is the Fourier transform.</figcaption> </figure> The same picture is clearer in three dimensions. Stack each wound copy of the signal along a third axis given by its winding frequency. Most slices are balanced loops centered on the axis; only at the true frequencies does the center of mass bulge outward.</p> The same winding viewed in 3D. Each slice is the signal wound at one frequency; the red centers of mass bulge out only at 2 and 3 Hz.</figcaption> </figure> That is the one-sentence summary of the entire transform: the DFT measures how well the signal lines up with a corkscrew of each frequency.</strong> The complex exponential $e^{-i 2\pi k n / N}$ is that corkscrew, and the sum is the alignment score. Everything that follows (the matrix form, the even/odd split, the butterflies) is just machinery for computing all $N$ of these alignment scores quickly.</p> The DFT as a Matrix</h2> Define the primitive root of unity</p> $$ \omega_N = e^{-i 2\pi / N}. $$</p> Then</p> $$ X_k = \sum_{n=0}^{N-1} x_n , \omega_N^{kn}. $$</p> Let $F_N$ be the $N \times N$ matrix with entries $(F_N)_{k,n} = \omega_N^{kn}$. Then the DFT is just a matrix-vector product:</p> $$ X = F_N x. $$</p> A matrix-vector product with a dense $N \times N$ matrix takes $O(N^2)$ multiplications and additions. For a one-second CD-quality audio frame with $N = 44{,}100$ samples that is about $2 \times 10^9$ operations per frame. Doable, but clearly wasteful if the matrix has structure.</p> It does. The matrix $F_N$ is extremely structured, because $\omega_N$ is a root of unity. The FFT is the algorithm that exploits that structure.</p> The FFT Idea: Split Into Even and Odd</h2> The Cooley-Tukey FFT is a divide-and-conquer algorithm. Assume $N$ is a power of two (we can always zero-pad to get there). Split the input sequence by index parity:</p> $$ \text{even part: } ; x_0, x_2, x_4, \ldots, x_{N-2}, $$</p> $$ \text{odd part: } ; x_1, x_3, x_5, \ldots, x_{N-1}. $$</p> Now rewrite $X_k$ by separating even and odd indices:</p> $$ X_k ;=; \sum_{m=0}^{N/2 - 1} x_{2m} , e^{-i 2\pi k (2m) / N} ;+; \sum_{m=0}^{N/2 - 1} x_{2m+1} , e^{-i 2\pi k (2m+1) / N}. $$</p> Pull out the extra factor in the odd sum:</p> $$ X_k ;=; \sum_{m=0}^{N/2 - 1} x_{2m} , e^{-i 2\pi k m / (N/2)} ;+; e^{-i 2\pi k / N} \sum_{m=0}^{N/2 - 1} x_{2m+1} , e^{-i 2\pi k m / (N/2)}. $$</p> Now look carefully at each sum. Each one is itself a DFT, of length $N/2$. Let</p> $$ E_k = \sum_{m=0}^{N/2 - 1} x_{2m} , e^{-i 2\pi k m / (N/2)}, \qquad O_k = \sum_{m=0}^{N/2 - 1} x_{2m+1} , e^{-i 2\pi k m / (N/2)}. $$</p> Then</p> $$ X_k = E_k + e^{-i 2\pi k / N} , O_k. $$</p> The quantity $e^{-i 2\pi k / N}$ is usually written</p> $$ W_N^k = e^{-i 2\pi k / N} $$</p> and called a twiddle factor</strong>. It is the bookkeeping that corrects for the shift between even and odd indices.</p> The Butterfly: Two Outputs for the Price of One</h2> We have computed $X_k$ in terms of $E_k$ and $O_k$ for $k = 0, 1, \ldots, N/2 - 1$. What about $k = N/2, N/2+1, \ldots, N-1$?</p> Use the periodicity of $E_k$ and $O_k$. Both are length-$N/2$ DFTs, so</p> $$ E_{k + N/2} = E_k, \qquad O_{k + N/2} = O_k. $$</p> The twiddle factor shifts by a sign:</p> $$ W_N^{k + N/2} = e^{-i 2\pi (k + N/2) / N} = e^{-i 2\pi k / N} e^{-i\pi} = -W_N^k. $$</p> Combining these,</p> $$ X_k = E_k + W_N^k , O_k, $$</p> $$ X_{k + N/2} = E_k - W_N^k , O_k. $$</p> That pair of equations is one butterfly</strong>. From two inputs $E_k$ and $O_k$ (and a precomputed twiddle), we produce two outputs $X_k$ and $X_{k+N/2}$ using one complex multiplication and two complex additions. We reuse work instead of redoing it.</p> At this point the whole algorithm is already visible. Compute two $N/2$-point DFTs. Combine them with $N/2$ butterflies to produce the $N$-point DFT. Recurse on each half.</p> Complexity: Why It Is $O(N \log N)$</h2> Let $T(N)$ be the number of operations to compute an $N$-point DFT using this recursion. We do:</p> Two DFTs of size $N/2$, each costing $T(N/2)$.</li> $N/2$ butterflies to combine them, each a constant amount of work. So $O(N)$ total for the combine step.</li> </ul> That gives the recurrence</p> $$ T(N) = 2 T(N/2) + cN, $$</p> with $T(1) = O(1)$, for some constant $c$.</p> Unroll the recurrence:</p> $$ T(N) ;=; 2T(N/2) + cN ;=; 4 T(N/4) + 2cN ;=; 8 T(N/8) + 3cN ;=; \ldots $$</p> After $\log_2 N$ levels of unrolling we reach $T(1)$, and each level contributes $cN$ work:</p> $$ T(N) = N \cdot T(1) + cN \log_2 N = O(N \log N). $$</p> Compare that to the naive DFT. For $N = 2^{20} \approx 10^6$:</p> $$ N^2 = 10^{12}, \qquad N \log_2 N \approx 2 \times 10^7. $$</p> Five orders of magnitude. That is the whole reason real-time audio processing, modern communications, and large-scale scientific computing are tractable.</p> A Worked Example: N = 4</h2> To see the recursion concretely, take $N = 4$ with input $x = (x_0, x_1, x_2, x_3)$.</p> The even part is $(x_0, x_2)$, and the odd part is $(x_1, x_3)$. Both are length-2 DFTs.</p> For a length-2 DFT with input $(a, b)$:</p> $$ Y_0 = a + b, \qquad Y_1 = a - b, $$</p> which is itself one butterfly with twiddle $W_2^0 = 1$.</p> So the two half-DFTs give</p> $$ E_0 = x_0 + x_2, \quad E_1 = x_0 - x_2, $$</p> $$ O_0 = x_1 + x_3, \quad O_1 = x_1 - x_3. $$</p> The twiddle factors at size $N=4$ are $W_4^0 = 1$ and $W_4^1 = e^{-i\pi/2} = -i$. Combining:</p> $$ X_0 = E_0 + W_4^0 , O_0 = (x_0 + x_2) + (x_1 + x_3), $$</p> $$ X_1 = E_1 + W_4^1 , O_1 = (x_0 - x_2) - i(x_1 - x_3), $$</p> $$ X_2 = E_0 - W_4^0 , O_0 = (x_0 + x_2) - (x_1 + x_3), $$</p> $$ X_3 = E_1 - W_4^1 , O_1 = (x_0 - x_2) + i(x_1 - x_3). $$</p> Four outputs, built from two pairs of butterflies. You can check by hand that these match $X_k = \sum_n x_n \omega_4^{kn}$ directly. This is the full FFT, in miniature.</p> Hand Calculation with Real Numbers</h2> The symbolic form is useful, but it is more convincing to watch the recursion run on actual numbers. Take the concrete input</p> $$ x = (x_0, x_1, x_2, x_3) = (1, 2, 3, 4). $$</p> Following the principle from the previous section step by step.</p> Step 1. Split by parity.</strong></p> $$ \text{even: } (x_0, x_2) = (1, 3), \qquad \text{odd: } (x_1, x_3) = (2, 4). $$</p> Step 2. Length-2 DFT of each half.</strong> Using $Y_0 = a + b$, $Y_1 = a - b$:</p> $$ E_0 = 1 + 3 = 4, \qquad E_1 = 1 - 3 = -2, $$</p> $$ O_0 = 2 + 4 = 6, \qquad O_1 = 2 - 4 = -2. $$</p> Step 3. Twiddle factors at $N = 4$.</strong></p> $$ W_4^0 = e^{0} = 1, \qquad W_4^1 = e^{-i\pi/2} = -i. $$</p> Step 4. Apply the butterfly equations.</strong> Using $X_k = E_k + W_4^k O_k$ and $X_{k+2} = E_k - W_4^k O_k$:</p> $$ X_0 = E_0 + W_4^0 , O_0 = 4 + (1)(6) = 10, $$</p> $$ X_1 = E_1 + W_4^1 , O_1 = -2 + (-i)(-2) = -2 + 2i, $$</p> $$ X_2 = E_0 - W_4^0 , O_0 = 4 - 6 = -2, $$</p> $$ X_3 = E_1 - W_4^1 , O_1 = -2 - (-i)(-2) = -2 - 2i. $$</p> So by hand,</p> $$ X = (10,; -2 + 2i,; -2,; -2 - 2i). $$</p> Step 5. Cross-check with code.</strong> Running the same input through numpy</code> and our from-scratch implementations should produce exactly the same vector:</p> import</span> numpy</span> as</span> np</span></span> </span> x</span> =</span> np</span>.</span>array</span>(</span>[</span>1</span>,</span> 2</span>,</span> 3</span>,</span> 4</span>]</span>,</span> dtype</span>=</span>complex</span>)</span></span> </span> print</span>(</span>"</span>numpy :</span>"</span>,</span> np</span>.</span>fft</span>.</span>fft</span>(</span>x</span>)</span>)</span></span> print</span>(</span>"</span>naive :</span>"</span>,</span> naive_dft</span>(</span>x</span>)</span>)</span></span> print</span>(</span>"</span>fft_rec:</span>"</span>,</span> fft_recursive</span>(</span>x</span>)</span>)</span></span></code></pre> Expected output (up to floating-point error):</p> numpy : [10.+0.j -2.+2.j -2.+0.j -2.-2.j]</span></span> naive : [10.+0.j -2.+2.j -2.+0.j -2.-2.j]</span></span> fft_rec: [10.+0.j -2.+2.j -2.+0.j -2.-2.j]</span></span></code></pre> Three different routes, one answer. The naive DFT sums all $N^2 = 16$ terms directly. The recursive FFT computes two length-2 DFTs and combines them with two butterflies, using far fewer operations. Both recover the same spectrum because they are computing the same mathematical object, just through different arithmetic paths.</p> This is the moment where the principle becomes tangible. Even/odd split, two half-DFTs, twiddle factor, butterfly. Four equations, one vector of outputs. Scale the same pattern recursively to $N = 8, 16, \ldots, 2^{20}$ and you have the full FFT.</p> From Scratch in Python: Naive DFT and Recursive FFT</h2> The fastest way to internalize this is to implement both and watch them agree.</p> Start with the naive DFT, a direct translation of the definition:</p> import</span> numpy</span> as</span> np</span></span> </span> def</span> naive_dft</span>(</span>x</span>)</span>:</span></span> x</span> =</span> np</span>.</span>asarray</span>(</span>x</span>,</span> dtype</span>=</span>complex</span>)</span></span> N</span> =</span> x</span>.</span>shape</span>[</span>0</span>]</span></span> n</span> =</span> np</span>.</span>arange</span>(</span>N</span>)</span></span> k</span> =</span> n</span>.</span>reshape</span>(</span>(</span>N</span>,</span> 1</span>)</span>)</span></span> M</span> =</span> np</span>.</span>exp</span>(</span>-</span>2</span>j</span> </span> np</span>.</span>pi</span> </span> k</span> </span> n</span> /</span> N</span>)</span></span> return</span> M</span> @</span> x</span></span></code></pre> Now the recursive Cooley-Tukey FFT. It assumes $N$ is a power of two. When $N$ drops to a small base case we hand off to the naive DFT, which is fine for tiny sizes:</p> def</span> fft_recursive</span>(</span>x</span>)</span>:</span></span> x</span> =</span> np</span>.</span>asarray</span>(</span>x</span>,</span> dtype</span>=</span>complex</span>)</span></span> N</span> =</span> x</span>.</span>shape</span>[</span>0</span>]</span></span> </span> if</span> N</span> &</span> (</span>N</span> -</span> 1</span>)</span>:</span></span> raise</span> ValueError</span>(</span>"</span>Length of input must be a power of 2.</span>"</span>)</span></span> </span> if</span> N</span> <=</span> 2</span>:</span></span> return</span> naive_dft</span>(</span>x</span>)</span></span> </span> X_even</span> =</span> fft_recursive</span>(</span>x</span>[</span>0</span>:</span>:</span>2</span>]</span>)</span></span> X_odd</span> =</span> fft_recursive</span>(</span>x</span>[</span>1</span>:</span>:</span>2</span>]</span>)</span></span> </span> k</span> =</span> np</span>.</span>arange</span>(</span>N</span> //</span> 2</span>)</span></span> twiddle</span> =</span> np</span>.</span>exp</span>(</span>-</span>2</span>j</span> </span> np</span>.</span>pi</span> </span> k</span> /</span> N</span>)</span></span> </span> return</span> np</span>.</span>concatenate</span>(</span>[</span>X_even</span> +</span> twiddle</span> </span> X_odd</span>,</span></span> X_even</span> -</span> twiddle</span> </span> X_odd</span>]</span>)</span></span></code></pre> Check agreement with numpy.fft.fft</code> on random input:</p> rng</span> =</span> np</span>.</span>random</span>.</span>default_rng</span>(</span>0</span>)</span></span> x</span> =</span> rng</span>.</span>standard_normal</span>(</span>1024</span>)</span></span> </span> X_naive</span> =</span> naive_dft</span>(</span>x</span>)</span></span> X_fft</span> =</span> fft_recursive</span>(</span>x</span>)</span></span> X_np</span> =</span> np</span>.</span>fft</span>.</span>fft</span>(</span>x</span>)</span></span> </span> print</span>(</span>np</span>.</span>allclose</span>(</span>X_naive</span>,</span> X_np</span>)</span>)</span></span> print</span>(</span>np</span>.</span>allclose</span>(</span>X_fft</span>,</span> X_np</span>)</span>)</span></span></code></pre> Both should print True</code> up to floating-point error. The three implementations compute the same mathematical object. The FFT just computes it faster.</p> A Real Sound-Wave Example</h2> Now the payoff. We construct a synthetic sound wave made of three tones, sample it, and recover those tones from the samples using the FFT.</p> import</span> numpy</span> as</span> np</span></span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span> </span> sample_rate</span> =</span> 8000</span></span> duration</span> =</span> 1.0</span></span> N</span> =</span> int</span>(</span>sample_rate</span> </span> duration</span>)</span></span> t</span> =</span> np</span>.</span>arange</span>(</span>N</span>)</span> /</span> sample_rate</span></span> </span> x</span> =</span> (</span>1.0</span> </span> np</span>.</span>sin</span>(</span>2</span> </span> np</span>.</span>pi</span> </span> 220</span> </span> t</span>)</span></span> +</span> 0.6</span> </span> np</span>.</span>sin</span>(</span>2</span> </span> np</span>.</span>pi</span> </span> 440</span> </span> t</span>)</span></span> +</span> 0.3</span> </span> np</span>.</span>sin</span>(</span>2</span> </span> np</span>.</span>pi</span> </span> 880</span> </span> t</span>)</span>)</span></span> </span> x</span> +=</span> 0.05</span> </span> np</span>.</span>random</span>.</span>default_rng</span>(</span>0</span>)</span>.</span>standard_normal</span>(</span>N</span>)</span></span></code></pre> The signal x</code> contains a strong $220$ Hz tone, a medium $440$ Hz tone, a weaker $880$ Hz tone, and a small amount of noise.</p> In the time domain the waveform looks periodic but visually messy. That is expected. Time domain is not the right view for this question.</p> A short window of the synthetic three-tone signal in the time domain.</figcaption> </figure> Now apply the FFT. Because N = 8000</code> is not a power of two, we either zero-pad up to the next power of two or use a mixed-radix FFT. For this note we will zero-pad, which is also a common practical trick to improve bin resolution:</p> def</span> next_pow2</span>(</span>n</span>)</span>:</span></span> return</span> 1</span> <<</span> (</span>n</span> -</span> 1</span>)</span>.</span>bit_length</span>(</span>)</span></span> </span> N_pad</span> =</span> next_pow2</span>(</span>N</span>)</span></span> x_pad</span> =</span> np</span>.</span>zeros</span>(</span>N_pad</span>)</span></span> x_pad</span>[</span>:</span>N</span>]</span> =</span> x</span></span> </span> X</span> =</span> fft_recursive</span>(</span>x_pad</span>)</span></span> magnitude</span> =</span> np</span>.</span>abs</span>(</span>X</span>)</span> /</span> N</span></span> freqs</span> =</span> np</span>.</span>arange</span>(</span>N_pad</span>)</span> </span> sample_rate</span> /</span> N_pad</span></span> </span> half</span> =</span> N_pad</span> //</span> 2</span></span> plt</span>.</span>plot</span>(</span>freqs</span>[</span>:</span>half</span>]</span>,</span> magnitude</span>[</span>:</span>half</span>]</span>)</span></span> plt</span>.</span>xlabel</span>(</span>"</span>Frequency (Hz)</span>"</span>)</span></span> plt</span>.</span>ylabel</span>(</span>"</span>Magnitude</span>"</span>)</span></span> plt</span>.</span>title</span>(</span>"</span>Magnitude spectrum of the synthetic sound wave</span>"</span>)</span></span> plt</span>.</span>grid</span>(</span>True</span>)</span></span> plt</span>.</span>show</span>(</span>)</span></span></code></pre> The spectrum shows three clean peaks, at $220$, $440$, and $880$ Hz, with heights roughly proportional to the amplitudes we put in. The noise spreads thinly across all other bins and stays small.</p> The DFT magnitude spectrum. The three input sinusoids appear as isolated peaks at 220, 440, and 880 Hz.</figcaption> </figure> That is the central picture. The time-domain view showed an oscillating mess. The frequency-domain view recovers exactly the three tones we synthesized, because that is what the DFT does. The FFT just made the computation fast enough to be practical on a full second of audio.</p> Naive DFT vs FFT: Timing</h2> To make the complexity story concrete, it helps to time both implementations on increasing input sizes. The naive DFT should grow quadratically, and the recursive FFT should grow close to linearly in $N \log N$:</p> import</span> time</span></span> </span> sizes</span> =</span> [</span>2</span>*</span>k</span> for</span> k</span> in</span> range</span>(</span>6</span>,</span> 13</span>)</span>]</span></span> naive_times</span> =</span> [</span>]</span></span> fft_times</span> =</span> [</span>]</span></span> </span> for</span> N</span> in</span> sizes</span>:</span></span> x</span> =</span> np</span>.</span>random</span>.</span>default_rng</span>(</span>0</span>)</span>.</span>standard_normal</span>(</span>N</span>)</span></span> </span> t0</span> =</span> time</span>.</span>perf_counter</span>(</span>)</span></span> naive_dft</span>(</span>x</span>)</span></span> naive_times</span>.</span>append</span>(</span>time</span>.</span>perf_counter</span>(</span>)</span> -</span> t0</span>)</span></span> </span> t0</span> =</span> time</span>.</span>perf_counter</span>(</span>)</span></span> fft_recursive</span>(</span>x</span>)</span></span> fft_times</span>.</span>append</span>(</span>time</span>.</span>perf_counter</span>(</span>)</span> -</span> t0</span>)</span></span> </span> plt</span>.</span>loglog</span>(</span>sizes</span>,</span> naive_times</span>,</span> marker</span>=</span>"</span>o</span>"</span>,</span> label</span>=</span>"</span>naive DFT</span>"</span>)</span></span> plt</span>.</span>loglog</span>(</span>sizes</span>,</span> fft_times</span>,</span> marker</span>=</span>"</span>o</span>"</span>,</span> label</span>=</span>"</span>recursive FFT</span>"</span>)</span></span> plt</span>.</span>xlabel</span>(</span>"</span>N</span>"</span>)</span></span> plt</span>.</span>ylabel</span>(</span>"</span>Time (s)</span>"</span>)</span></span> plt</span>.</span>title</span>(</span>"</span>Runtime: naive DFT vs recursive FFT</span>"</span>)</span></span> plt</span>.</span>legend</span>(</span>)</span></span> plt</span>.</span>grid</span>(</span>True</span>,</span> which</span>=</span>"</span>both</span>"</span>)</span></span> plt</span>.</span>show</span>(</span>)</span></span></code></pre> On a log-log plot, the naive DFT has slope close to $2$ and the recursive FFT has slope close to $1$. That is the $O(N^2)$ versus $O(N \log N)$ gap, visualized directly.</p> Empirical runtime comparison. The naive DFT grows quadratically while the recursive FFT grows close to linearly in N log N.</figcaption> </figure> Sampling, Nyquist, and Aliasing</h2> A few practical points are worth making explicit, because they trip people up in real audio code.</p> The highest frequency the DFT can represent is the Nyquist frequency $f_s / 2$. At sample rate $8000$ Hz, that is $4000$ Hz. Any signal component above that frequency does not simply vanish. It aliases down into a lower bin and corrupts the spectrum. That is why audio is low-pass filtered before sampling.</p> For real-valued inputs, the spectrum is conjugate-symmetric: $X_{N-k} = \overline{X_k}$. So the second half of the DFT output carries no new information. In practice we only plot bins $0$ through $N/2$.</p> The frequency resolution of the DFT is $f_s / N$. Longer windows give finer frequency resolution but worse time resolution. This trade-off is exactly why short-time Fourier transforms and wavelets exist.</p> Windowing and Spectral Leakage</h2> If the signal frequency does not land exactly on a DFT bin, its energy spreads into neighboring bins. This is called spectral leakage, and it is a property of the DFT, not a bug. Leakage is reduced by multiplying the signal by a smooth window function</strong> before the transform:</p> window</span> =</span> np</span>.</span>hanning</span>(</span>N</span>)</span></span> x_windowed</span> =</span> x</span> </span> window</span></span></code></pre> A Hann window tapers the signal smoothly to zero at both ends, which suppresses the discontinuity that causes leakage. The trade-off is a slight widening of each spectral peak. For most audio applications this is a good deal.</p> Real Input: A Slightly Faster Variant</h2> When the input is real, numpy.fft.rfft</code> returns only the non-redundant half of the spectrum and is about twice as fast as the full complex FFT. In from-scratch code the same idea can be implemented by exploiting the conjugate symmetry directly. For production use, numpy.fft.rfft</code> or any mature FFT library (FFTW, PocketFFT, cuFFT) is almost always the right choice. The recursive implementation above is for understanding, not for speed.</p> The Big Picture</h2> At this point the whole thing should feel like one continuous argument, not three unrelated algorithms.</p> We started with a sound wave in the time domain and asked which frequencies it contains. Sampling gave us a finite sequence $x_n$. Projecting that sequence onto a basis of complex exponentials gave the DFT:</p> $$ X_k = \sum_{n=0}^{N-1} x_n , e^{-i 2\pi k n / N}. $$</p> The DFT is a linear map, so it can be written as a matrix product $X = F_N x$. Computed naively, that is $O(N^2)$.</p> We then exploited the structure of $F_N$. The roots of unity satisfy a recursion, so the $N$-point DFT splits into two $N/2$-point DFTs combined by $N/2$ butterflies. That gives the recurrence $T(N) = 2T(N/2) + O(N)$ and the complexity $O(N \log N)$.</p> Finally, we tested the whole pipeline on a synthetic sound wave and recovered the three input tones as three clean peaks in the magnitude spectrum.</p> So in the end, the DFT is the definition and the FFT is the algorithm. The DFT says what we want. The FFT says how to get it quickly. The exponential keeps showing up for the same reason it does in logistic and softmax regression: it is the right basis when the underlying object has multiplicative structure. Roots of unity are multiplicative. Splitting them in half, again and again, is exactly what turns $N^2$ into $N \log N$.</p> If I keep going with this, the next stops are the short-time Fourier transform and spectrograms, and after that the jump from Fourier bases to wavelets. Corrections and better examples are welcome in the comments.</p> Softmax Regression Explained from Logistic Regression 2026-04-19T00:00:00+00:00 Softmax regression is what logistic regression grows into when there are more than two classes, and logistic regression itself only really makes sense once you understand odds. So instead of starting at the softmax formula, this note starts all the way back at odds and walks forward:</p> $$ \text{probability} \to \text{odds} \to \text{log-odds} \to \text{logistic regression} \to \text{softmax regression}. $$</p> By the end, the exponential should not feel like a trick anymore. It is just the machine that turns unrestricted linear scores into positive ratios.</p> If you want the full runnable walkthrough, the companion notebook lives on Kaggle:</p> https://www.kaggle.com/code/lyhatt/softmax-from-logistic-regression</a></p> That notebook was assisted by AI tools. This note is the written version.</p> Start with Probability, Then Ask for More Structure</h2> Suppose we have a binary outcome. Let</p> $$ p = P(y=1 \mid x) $$</p> be the probability that the positive class occurs given the input $x$.</p> Probability is easy to interpret, but it is not yet the best thing to model. A slightly more structured object is the odds</strong>:</p> $$ \text{odds} = \frac{p}{1-p}. $$</p> The odds tell us how much more likely the positive event is than the negative one.</p> Some quick examples help. If $p = 0.5$, then the odds are $1$. If $p = 0.8$, then the odds are $\frac{0.8}{0.2} = 4$. If $p = 0.2$, then the odds are $\frac{0.2}{0.8} = 0.25$. So odds greater than $1$ favor the positive class, odds less than $1$ favor the negative class, and odds equal to $1$ mean the two classes are balanced.</p> This is already useful, but there is a problem. Odds are always positive, so they live on $(0,\infty)$. A plain linear model</p> $$ w^T x + b $$</p> lives on the whole real line, from $-\infty$ to $\infty$. Those two spaces do not match well. If we want a linear model, odds are the wrong coordinates.</p> The Core Move: Take Logarithms</h2> To fix this, we take the logarithm of the odds:</p> $$ \log \frac{p}{1-p}. $$</p> This quantity is the log-odds</strong>, also called the logit</strong>.</p> Why is this useful? Because odds are constrained to be positive, but log-odds can be any real number. A logit of $0$ corresponds to $p=0.5$, positive logits correspond to probabilities above $0.5$, and negative logits correspond to probabilities below $0.5$. This is the key move. The logarithm converts a positive ratio into a quantity that can now range over the whole real line. That means a linear model suddenly fits naturally:</p> $$ \log \frac{p}{1-p} = w^T x + b. $$</p> This is logistic regression in its cleanest form. The model says: not that the probability is linear, but that the log-odds</strong> are linear.</p> Now Solve for the Probability</h2> Now let us solve for $p$ carefully. Start with</p> $$ \log \frac{p}{1-p} = w^T x + b. $$</p> Exponentiate both sides:</p> $$ \frac{p}{1-p} = e^{w^T x + b}. $$</p> This is where the exponential enters the story, and it is doing real mathematical work. It appears because it is the inverse of the logarithm, because it guarantees the odds stay positive, and because it converts an unrestricted real-valued score into a valid positive ratio.</p> Now multiply both sides by $(1-p)$:</p> $$ p = (1-p)e^{w^T x + b}. $$</p> Expand the right-hand side:</p> $$ p = e^{w^T x + b} - pe^{w^T x + b}. $$</p> Collect the $p$ terms:</p> $$ p + pe^{w^T x + b} = e^{w^T x + b}. $$</p> Factor out $p$:</p> $$ p\left(1 + e^{w^T x + b}\right) = e^{w^T x + b}. $$</p> So</p> $$ p = \frac{e^{w^T x + b}}{1 + e^{w^T x + b}}. $$</p> Divide numerator and denominator by $e^{w^T x + b}$:</p> $$ p = \frac{1}{1 + e^{-(w^T x + b)}}. $$</p> This is the sigmoid form of logistic regression:</p> $$ p = \sigma(w^T x + b). $$</p> This is the full derivation. The sigmoid is not a decorative choice. It is what falls out when a linear model is placed on the log-odds.</p> What the Sigmoid Is Really Doing</h2> The sigmoid takes a real-valued score and maps it to a number between $0$ and $1$:</p> $$ \sigma(z) = \frac{1}{1+e^{-z}}. $$</p> Here $z = w^T x + b$ is often called the logit</strong>.</p> The interpretation is immediate once you remember that $z$ is a logit. If $z \gg 0$, then $\sigma(z)$ is close to $1$. If $z = 0$, then $\sigma(z)=0.5$. If $z \ll 0$, then $\sigma(z)$ is close to $0$. So the sigmoid is the map from linear evidence to probability. That is why logistic regression has the familiar S-curve.</p> The sigmoid curve appears after solving the log-odds equation for the probability.</figcaption> </figure> The Model Needs a Learning Signal</h2> Once the model outputs probabilities, the next question is: what objective should it optimize? Logistic regression uses the Bernoulli likelihood. For one example with label $y \in {0,1}$ and predicted probability $\hat{y}$,</p> $$ P(y \mid x) = \hat{y}^y(1-\hat{y})^{1-y}. $$</p> Taking logs gives the log-likelihood:</p> $$ \log P(y \mid x) = y \log \hat{y} + (1-y)\log(1-\hat{y}). $$</p> For $m$ observations, the negative average log-likelihood becomes the binary cross-entropy:</p> $$ L = -\frac{1}{m} \sum_{i=1}^m \left[y^{(i)} \log \hat{y}^{(i)} + (1-y^{(i)})\log(1-\hat{y}^{(i)})\right]. $$</p> This loss is worth staring at for a moment. It tells you exactly what the model fears: confident mistakes.</p> Binary cross-entropy for the two possible true labels.</figcaption> </figure> That is the right behavior for a probabilistic model. If it is unsure, the penalty is moderate. If it is confidently wrong, the penalty should explode.</p> The Optimization Is Still Just Gradient Descent</h2> Once we define the loss, the training rule is still gradient descent. The gradients are</p> $$ \nabla_w L = \frac{1}{m}X^T(\hat{y}-y), \qquad \nabla_b L = \frac{1}{m}\sum_{i=1}^{m}(\hat{y}^{(i)} - y^{(i)}). $$</p> So the update step is</p> $$ w \leftarrow w - \alpha \nabla_w L, \qquad b \leftarrow b - \alpha \nabla_b L. $$</p> This is worth emphasizing. Logistic regression sounds like a special model class, but algorithmically it is still simple: define probabilities, define a loss, differentiate, and descend.</p> A Real Binary Example</h2> To make the binary story concrete, we use the Breast Cancer Wisconsin dataset. It has 569 samples, 30 numerical features, and two target classes: 0</code> for malignant and 1</code> for benign.</p> Before fitting anything, it helps to get a feel for the data itself.</p> Class balance and a two-feature raw view of the Breast Cancer dataset.</figcaption> </figure> This is just the intuition view. The real training run uses all standardized features, not only the two shown here.</p> Below is a compact from-scratch implementation of logistic regression with gradient descent:</p> import</span> numpy</span> as</span> np</span></span> </span> def</span> sigmoid</span>(</span>z</span>)</span>:</span></span> return</span> 1.0</span> /</span> (</span>1.0</span> +</span> np</span>.</span>exp</span>(</span>-</span>z</span>)</span>)</span></span> </span> def</span> binary_cross_entropy</span>(</span>y_true</span>,</span> y_prob</span>)</span>:</span></span> eps</span> =</span> 1e-12</span></span> y_prob</span> =</span> np</span>.</span>clip</span>(</span>y_prob</span>,</span> eps</span>,</span> 1</span> -</span> eps</span>)</span></span> return</span> -</span>np</span>.</span>mean</span>(</span>y_true</span> </span> np</span>.</span>log</span>(</span>y_prob</span>)</span> +</span> (</span>1</span> -</span> y_true</span>)</span> </span> np</span>.</span>log</span>(</span>1</span> -</span> y_prob</span>)</span>)</span></span> </span> def</span> train_binary_logistic</span>(</span>X</span>,</span> y</span>,</span> learning_rate</span>=</span>0.1</span>,</span> epochs</span>=</span>1500</span>)</span>:</span></span> n_samples</span>,</span> n_features</span> =</span> X</span>.</span>shape</span></span> weights</span> =</span> np</span>.</span>zeros</span>(</span>n_features</span>)</span></span> bias</span> =</span> 0.0</span></span> history</span> =</span> [</span>]</span></span> </span> for</span> _</span> in</span> range</span>(</span>epochs</span>)</span>:</span></span> logits</span> =</span> X</span> @</span> weights</span> +</span> bias</span></span> probs</span> =</span> sigmoid</span>(</span>logits</span>)</span></span> error</span> =</span> probs</span> -</span> y</span></span> grad_w</span> =</span> X</span>.</span>T</span> @</span> error</span> /</span> n_samples</span></span> grad_b</span> =</span> np</span>.</span>mean</span>(</span>error</span>)</span></span> weights</span> -=</span> learning_rate</span> </span> grad_w</span></span> bias</span> -=</span> learning_rate</span> </span> grad_b</span></span> history</span>.</span>append</span>(</span>binary_cross_entropy</span>(</span>y</span>,</span> probs</span>)</span>)</span></span> </span> return</span> weights</span>,</span> bias</span>,</span> np</span>.</span>array</span>(</span>history</span>)</span></span></code></pre> The training loss falls in the way we expect:</p> Training loss for from-scratch logistic regression.</figcaption> </figure> If we restrict ourselves to two features, we can also draw the probability surface directly:</p> A two-feature probability surface for logistic regression.</figcaption> </figure> That is enough for the binary story. The point is not to pile up diagnostics. The point is to see that logistic regression turns a linear score into a probability, and then learns that mapping through gradient descent.</p> The Jump to Many Classes</h2> Now comes the real question: what changes when there are not two classes, but many?</p> In binary logistic regression, we can think of the model as comparing two outcomes. The odds are</p> $$ \frac{p}{1-p}. $$</p> That ratio is already a comparison between classes. Softmax keeps the same spirit, but scales it from two classes to $K$ classes.</p> One useful way to think about this is to stop thinking in terms of labels first and think in terms of score comparisons</strong>.</p> Imagine an image classifier. A certain image might look a bit like a car, somewhat like a truck, and definitely not like a frog. In that situation, the model does not begin with probabilities. It begins with raw scores. Maybe the car score is high, the truck score is also fairly high, and the frog score is very low. That is already a meaningful picture. "Car versus truck" is a hard distinction because those classes can share visual structure. "Car versus frog" is easier because the scores should separate much more aggressively.</p> Softmax takes exactly this kind of score vector and says: fine, now turn those relative preferences into a probability distribution.</p> Suppose each class $k$ gets its own score</p> $$ z_k. $$</p> If we want a multiclass probability model, the outputs should satisfy three requirements. Each probability must be positive, all probabilities must sum to $1$, and a larger score should correspond to a larger probability. The exponential is the natural choice here because $e^{z_k}$ is always positive, because if $z_a > z_b$ then $e^{z_a} > e^{z_b}$, and because exponentials turn unrestricted scores into positive quantities that can be normalized.</p> So we first define unnormalized class weights:</p> $$ \tilde{p}_k = e^{z_k}. $$</p> These are not probabilities yet, because they do not sum to $1$. So we normalize them:</p> $$ p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}. $$</p> That normalization step is softmax.</p> Why Exponentials Keep Showing Up</h2> At this point the exponential should no longer feel mysterious.</p> For logistic regression, we start from log-odds, exponentiation recovers the odds, and the final algebra gives the sigmoid probability. For softmax regression, we start from a score for each class, exponentiation turns each score into a positive class weight, and normalization turns those positive weights into a probability distribution. It is the same structural tool in both models: take unrestricted scores, convert them into positive quantities, and only then interpret them probabilistically.</p> There is also an intuitive reason practitioners like this view. A difference in score should matter more when it is decisive and less when two classes are nearly tied. Exponentials amplify those differences in a smooth way. If "car" scores only slightly above "truck", the probabilities stay competitive. If "car" scores far above "frog", the probability mass shifts much more aggressively.</p> Softmax Actually Contains Logistic Regression</h2> Softmax is the multiclass extension of logistic regression, and we can show that mathematically.</p> Take the two-class case with scores $z_0$ and $z_1$. Then softmax gives</p> $$ p_1 = \frac{e^{z_1}}{e^{z_0} + e^{z_1}}. $$</p> Factor out $e^{z_1}$ in the denominator:</p> $$ p_1 = \frac{e^{z_1}}{e^{z_1}\left(e^{z_0-z_1}+1\right)} = \frac{1}{1 + e^{z_0-z_1}}. $$</p> Equivalently,</p> $$ p_1 = \frac{1}{1 + e^{-(z_1-z_0)}}. $$</p> This is exactly a sigmoid in the score difference $z_1-z_0$.</p> That is the bridge. Logistic regression is the binary special case, and softmax regression is the multiclass generalization.</p> The Multiclass Loss Is the Same Idea Again</h2> In the multiclass setting, the true label is represented as a one-hot vector. If $Y$ is the one-hot target matrix and $P$ is the predicted softmax probability matrix, then the multiclass cross-entropy is</p> $$ L = -\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K} y_k^{(i)} \log p_k^{(i)}. $$</p> This is the same negative log-likelihood idea again, just written for many classes instead of two.</p> The gradient keeps a remarkably clean form:</p> $$ \nabla_W L = \frac{1}{m}X^T(P-Y), \qquad \nabla_b L = \frac{1}{m}\sum_{i=1}^{m}(P^{(i)} - Y^{(i)}). $$</p> So the optimization rule is still gradient descent:</p> $$ W \leftarrow W - \alpha \nabla_W L, \qquad b \leftarrow b - \alpha \nabla_b L. $$</p> So the whole picture survives the transition almost unchanged. The binary weight vector becomes a class-wise weight matrix, and everything else follows.</p> A Real Multiclass Example</h2> For the multiclass example, we use the Digits dataset. This is much more appropriate for softmax because it contains ten classes instead of two.</p> Each digit image is an $8 \times 8$ grayscale grid, flattened into a 64-dimensional vector.</p> Before fitting the model, it is worth seeing both raw samples and class averages:</p> Raw digits samples and average digit images by class.</figcaption> </figure> Here is a compact from-scratch softmax regression implementation:</p> import</span> numpy</span> as</span> np</span></span> </span> def</span> softmax</span>(</span>logits</span>)</span>:</span></span> shifted</span> =</span> logits</span> -</span> np</span>.</span>max</span>(</span>logits</span>,</span> axis</span>=</span>1</span>,</span> keepdims</span>=</span>True</span>)</span></span> exp_logits</span> =</span> np</span>.</span>exp</span>(</span>shifted</span>)</span></span> return</span> exp_logits</span> /</span> np</span>.</span>sum</span>(</span>exp_logits</span>,</span> axis</span>=</span>1</span>,</span> keepdims</span>=</span>True</span>)</span></span> </span> def</span> multiclass_cross_entropy</span>(</span>y_one_hot</span>,</span> probs</span>)</span>:</span></span> eps</span> =</span> 1e-12</span></span> probs</span> =</span> np</span>.</span>clip</span>(</span>probs</span>,</span> eps</span>,</span> 1</span> -</span> eps</span>)</span></span> return</span> -</span>np</span>.</span>mean</span>(</span>np</span>.</span>sum</span>(</span>y_one_hot</span> </span> np</span>.</span>log</span>(</span>probs</span>)</span>,</span> axis</span>=</span>1</span>)</span>)</span></span> </span> def</span> train_softmax_regression</span>(</span>X</span>,</span> y</span>,</span> learning_rate</span>=</span>0.1</span>,</span> epochs</span>=</span>3000</span>)</span>:</span></span> n_samples</span>,</span> n_features</span> =</span> X</span>.</span>shape</span></span> n_classes</span> =</span> len</span>(</span>np</span>.</span>unique</span>(</span>y</span>)</span>)</span></span> weights</span> =</span> np</span>.</span>zeros</span>(</span>(</span>n_features</span>,</span> n_classes</span>)</span>)</span></span> bias</span> =</span> np</span>.</span>zeros</span>(</span>n_classes</span>)</span></span> y_one_hot</span> =</span> np</span>.</span>eye</span>(</span>n_classes</span>)</span>[</span>y</span>]</span></span> history</span> =</span> [</span>]</span></span> </span> for</span> _</span> in</span> range</span>(</span>epochs</span>)</span>:</span></span> logits</span> =</span> X</span> @</span> weights</span> +</span> bias</span></span> probs</span> =</span> softmax</span>(</span>logits</span>)</span></span> error</span> =</span> probs</span> -</span> y_one_hot</span></span> grad_w</span> =</span> X</span>.</span>T</span> @</span> error</span> /</span> n_samples</span></span> grad_b</span> =</span> np</span>.</span>mean</span>(</span>error</span>,</span> axis</span>=</span>0</span>)</span></span> weights</span> -=</span> learning_rate</span> </span> grad_w</span></span> bias</span> -=</span> learning_rate</span> </span> grad_b</span></span> history</span>.</span>append</span>(</span>multiclass_cross_entropy</span>(</span>y_one_hot</span>,</span> probs</span>)</span>)</span></span> </span> return</span> weights</span>,</span> bias</span>,</span> np</span>.</span>array</span>(</span>history</span>)</span></span></code></pre>How to Visualize Softmax Honestly</h2> Sigmoid has a universal curve. Softmax does not.</p> So the honest softmax visualization is not a fake one-dimensional curve. It is a sample-level transformation: start from a real digit image, compute one raw logit per class, and then exponentiate and normalize into probabilities. That is exactly what the next figure shows:</p> A real digit mapped into logits and then into softmax probabilities.</figcaption> </figure> That is the mental model to keep: softmax takes a vector of scores and turns it into a probability distribution.</p> In other words, softmax is less about drawing one magical curve and more about resolving competitions among classes. Some competitions are close, like car versus truck. Some are not, like car versus frog. The probability vector tells us how that competition ended for one specific input.</p> The training loss on Digits decreases as expected:</p> Training loss for from-scratch softmax regression on Digits.</figcaption> </figure> That is enough for the multiclass side as well. Once the reader sees logits become probabilities on a real sample, the central idea has already landed. The remaining plots are useful in a notebook, but they are not necessary in the main essay.</p> A Few Practical Tradeoffs</h2> This is also a good place to be honest about what logistic regression and softmax regression can and cannot do. They are simple, fast, interpretable, and they produce probabilities directly, which is often exactly what we want. They also work surprisingly well when the classes are close to linearly separable in feature space. But the tradeoff is clear: if the real boundary is highly nonlinear, these models will struggle, the quality of the features matters a great deal, and multiclass confusion often comes from genuinely similar classes rather than from a bug in the formula.</p> This is also where people usually start reaching for nonlinear models. In deep learning, nonlinear functions become popular precisely because they can build more complex decision boundaries than a single linear logit layer can express. They are often used as activation functions inside deeper networks, and that added nonlinearity is what lets the model represent more complicated structure. But that is really another topic. Here the point is to understand the clean linear baseline first.</p> That is why these models are such good teaching models and such good baselines. When they succeed, they do so for understandable reasons. When they fail, they usually fail in ways that teach us something about the data.</p> The Big Picture</h2> At this point the whole thing should feel like one continuous argument, not three unrelated formulas.</p> We started with probability, then moved to odds:</p> $$ \frac{p}{1-p}. $$</p> We took logarithms to obtain log-odds:</p> $$ \log \frac{p}{1-p}. $$</p> We modeled the log-odds linearly and solved for the probability, which produced the sigmoid and logistic regression.</p> Then we asked what happens when there are many classes instead of two. The same need appears again: we must turn unrestricted scores into valid positive ratios, and then normalize them into probabilities. That requirement leads directly to the softmax formula.</p> So logistic regression is a model for binary log-odds, and softmax regression is the same idea with more classes. The exponential appears because it converts linear scores into positive ratios, and gradient descent is the engine underneath both. Learn them together; separately they both look more arbitrary than they are.</p> Natural continuations from here: cross-entropy from maximum likelihood, and the step from softmax regression to a neural network, which is just softmax with learned features. Corrections welcome in the comments.</p> Statistical Testing in Regression 2026-02-20T00:00:00+00:00 This note covers hypothesis testing for regression coefficients and model terms, with emphasis on what p-values can and cannot tell you. The practical problem is making defensible decisions under uncertainty without overclaiming. I focus on test setup, assumptions, and interpretation boundaries.</p> In the previous post</a>, we constructed confidence intervals for regression coefficients. Confidence intervals and hypothesis tests are two sides of the same coin. In this post, we formalize the hypothesis testing framework for regression coefficients.</p> Hypothesis Testing for Individual Coefficients</h2> The most common test in regression asks whether a particular predictor has a significant linear effect on the response. For coefficient $\beta_j$, the hypotheses are:</p> $$H_0: \beta_j = 0 \quad \text{(the predictor has no effect)},$$ $$H_1: \beta_j \neq 0 \quad \text{(the predictor has an effect)}.$$</p> This is a two-sided test. One-sided alternatives ($H_1: \beta_j > 0$ or $H_1: \beta_j < 0$) are also possible when the direction of the effect is of interest.</p> The t-Test Statistic</h2> Under $H_0: \beta_j = 0$, the test statistic is:</p> $$t = \frac{\hat{\beta}_j - 0}{SE(\hat{\beta}_j)} = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)}.$$</p> Under the null hypothesis and the normality assumption, this statistic follows a t-distribution:</p> $$t \sim t_{n-p},$$</p> where $n - p$ is the residual degrees of freedom.</p> For Simple Linear Regression</h3> Testing $H_0: \beta_1 = 0$ (the slope is zero):</p> $$t = \frac{\hat{\beta}_1}{SE(\hat{\beta}_1)} = \frac{\hat{\beta}_1}{\sqrt{MSE/S_{xx}}} = \frac{S_{xy}/S_{xx}}{\sqrt{MSE/S_{xx}}} = \frac{S_{xy}}{\sqrt{MSE \cdot S_{xx}}}.$$</p> This statistic follows $t_{n-2}$ under $H_0$.</p> Testing $H_0: \beta_0 = 0$ (the intercept is zero):</p> $$t = \frac{\hat{\beta}_0}{SE(\hat{\beta}_0)} = \frac{\hat{\beta}_0}{\sqrt{MSE(1/n + \bar{x}^2/S_{xx})}}.$$</p> For Multiple Regression</h3> Testing $H_0: \beta_i = 0$ for any coefficient $\beta_i$:</p> $$t = \frac{\hat{\beta}_i}{SE(\hat{\beta}_i)} = \frac{\hat{\beta}_i}{\sqrt{MSE \cdot C_{ii}}},$$</p> where $C_{ii}$ is the appropriate diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</p> Decision Rule</h2> For a significance level $\alpha$ (commonly $\alpha = 0.05$):</p> Reject $H_0$</strong> if $\|t\| > t_{\alpha/2, n-p}$.</p> Fail to reject $H_0$</strong> if $\|t\| \leq t_{\alpha/2, n-p}$.</p> Rejecting $H_0$ means we have sufficient evidence that $\beta_j \neq 0$, suggesting that the predictor $x_j$ has a statistically significant effect on $y$.</p> P-Values</h2> The p-value provides a continuous measure of the evidence against $H_0$. For a two-sided test:</p> $$p\text{-value} = 2 \cdot P(T > \|t_{\text{obs}}\|),$$</p> where $T \sim t_{n-p}$ and $t_{\text{obs}}$ is the observed test statistic.</p> Interpretation:</p> The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming $H_0$ is true.</li> A small p-value (typically $< 0.05$) provides evidence against $H_0$.</li> A large p-value does not prove $H_0$ is true; it simply means the data do not provide strong evidence against it.</li> </ul> The decision rule using p-values is equivalent to using critical values: reject $H_0$ if $p\text{-value} < \alpha$.</p> Testing Against a Non-Zero Value</h2> More generally, we can test $H_0: \beta_j = \beta_{j,0}$ for any hypothesized value $\beta_{j,0}$:</p> $$t = \frac{\hat{\beta}_j - \beta_{j,0}}{SE(\hat{\beta}_j)}.$$</p> The test for $\beta_j = 0$ is simply the special case where $\beta_{j,0} = 0$.</p> Connection to Confidence Intervals</h2> There is a direct equivalence between hypothesis tests and confidence intervals:</p> Reject</strong> $H_0: \beta_j = \beta_{j,0}$ at level $\alpha$ if and only if $\beta_{j,0}$ falls outside the $(1 - \alpha) \times 100%$ confidence interval for $\beta_j$.</li> Fail to reject</strong> $H_0$ if and only if $\beta_{j,0}$ falls inside the confidence interval.</li> </ul> This means that when we check whether a 95% confidence interval for $\beta_1$ contains zero, we are implicitly performing a t-test at the $\alpha = 0.05$ level.</p> Testing the Correlation</h2> For simple linear regression, testing $H_0: \beta_1 = 0$ is equivalent to testing $H_0: \rho = 0$ (the population correlation is zero). This is because $\hat{\beta}_1 = r \cdot S_y/S_x$, and $\hat{\beta}_1 = 0$ if and only if $r = 0$.</p> The test statistic can also be written in terms of $r$:</p> $$t = r\sqrt{\frac{n - 2}{1 - r^2}},$$</p> which follows $t_{n-2}$ under $H_0: \rho = 0$.</p> Practical vs. Statistical Significance</h2> A statistically significant result (small p-value) does not necessarily mean the result is practically important. With a large sample size, even a tiny effect can be statistically significant because the standard error shrinks with $n$.</p> Conversely, a non-significant result does not mean the effect is zero; it may reflect insufficient sample size (low statistical power).</p> When interpreting regression results, consider both:</p> Statistical significance</strong>: Is the p-value small enough to reject $H_0$?</li> Practical significance</strong>: Is the estimated effect $\hat{\beta}_j$ large enough to matter in the context of the problem?</li> </ol> The confidence interval is particularly helpful here because it shows both the direction and the plausible range of the effect size.</p> Multiple Testing Considerations</h2> When testing multiple coefficients in a regression model, each at level $\alpha$, the probability of at least one false rejection increases. If you test $k$ independent hypotheses at $\alpha = 0.05$, the family-wise error rate is approximately $1 - (1 - \alpha)^k$.</p> Common corrections include:</p> Bonferroni correction</strong>: Test each hypothesis at $\alpha/k$.</li> Holm-Bonferroni method</strong>: A sequential version that is less conservative.</li> False Discovery Rate (FDR) control</strong>: Controls the expected proportion of false rejections.</li> </ul> In regression, the overall model significance is typically assessed using the F-test (covered in the next post), which tests all predictors simultaneously.</p> Summary</h2> In this post, we covered hypothesis testing for regression coefficients:</p> The t-statistic $t = \hat{\beta}_j / SE(\hat{\beta}_j)$ follows $t_{n-p}$ under $H_0: \beta_j = 0$.</li> Reject $H_0$ when $\|t\| > t_{\alpha/2, n-p}$ or equivalently when $p\text{-value} < \alpha$.</li> For $\beta_1$ in SLR: $t = S_{xy}/\sqrt{MSE \cdot S_{xx}}$ with $n - 2$ degrees of freedom.</li> Testing $\beta_1 = 0$ is equivalent to testing $\rho = 0$.</li> Confidence intervals and hypothesis tests provide equivalent information.</li> Always consider practical significance alongside statistical significance.</li> </ul> In the next post, we will examine the ANOVA framework, which tests the overall significance of the regression model using the F-test.</p> Confidence Intervals for Regression Coefficients 2026-02-13T00:00:00+00:00 This note shows how confidence intervals for regression coefficients are constructed and how to interpret them correctly. The core issue is avoiding binary thinking around single-point estimates. The objective is to connect standard errors, critical values, and interval interpretation in a reproducible way.</p> In the previous post</a>, we derived the standard errors of the regression coefficients. Standard errors tell us about the precision of our estimates, but they do not directly give us a range of plausible values for the true parameters. Confidence intervals do exactly that.</p> The Sampling Distribution of $\hat{\beta}_j$</h2> Under the normality assumption $\varepsilon_i \sim N(0, \sigma^2)$, the OLS estimators are also normally distributed:</p> $$\hat{\beta}_j \sim N\left(\beta_j, \sigma^2 C_{jj}\right),$$</p> where $C_{jj}$ is the appropriate diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</p> Standardizing gives:</p> $$\frac{\hat{\beta}_j - \beta_j}{\sigma\sqrt{C_{jj}}} \sim N(0, 1).$$</p> However, $\sigma^2$ is unknown and we replace it with $MSE$. This introduces additional uncertainty, and the resulting distribution changes from a standard normal to a t-distribution:</p> $$\frac{\hat{\beta}_j - \beta_j}{SE(\hat{\beta}_j)} \sim t_{n - p},$$</p> where $SE(\hat{\beta}_j) = \sqrt{MSE \cdot C_{jj}}$ and $n - p$ is the degrees of freedom.</p> Confidence Interval Formula</h2> A $(1 - \alpha) \times 100%$ confidence interval for $\beta_j$ is:</p> $$\hat{\beta}_j \pm t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_j),$$</p> where $t_{\alpha/2, n-p}$ is the critical value from the t-distribution with $n - p$ degrees of freedom such that $P(\|T\| > t_{\alpha/2, n-p}) = \alpha$.</p> Equivalently, the interval is:</p> $$\left[\hat{\beta}_j - t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_j), \quad \hat{\beta}_j + t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_j)\right].$$</p> Confidence Interval for $\beta_1$ (Simple Linear Regression)</h2> For the slope in simple linear regression, $n - p = n - 2$:</p> $$\hat{\beta}_1 \pm t_{\alpha/2,, n-2} \cdot SE(\hat{\beta}_1),$$</p> where:</p> $$SE(\hat{\beta}_1) = \sqrt{\frac{MSE}{S_{xx}}}.$$</p> Expanding fully:</p> $$\hat{\beta}_1 \pm t_{\alpha/2,, n-2} \cdot \sqrt{\frac{SSE}{(n-2) \cdot S_{xx}}}.$$</p> This interval is centered at the OLS estimate $\hat{\beta}_1$ and its width depends on three factors:</p> Confidence level</strong> ($1 - \alpha$): Higher confidence means a wider interval (larger $t_{\alpha/2}$).</li> Residual variability</strong> ($MSE$): More noise means a wider interval.</li> Predictor variability</strong> ($S_{xx}$): More spread in $x$ means a narrower interval.</li> </ol> Confidence Interval for $\beta_0$ (Simple Linear Regression)</h2> For the intercept:</p> $$\hat{\beta}_0 \pm t_{\alpha/2,, n-2} \cdot SE(\hat{\beta}_0),$$</p> where:</p> $$SE(\hat{\beta}_0) = \sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right)}.$$</p> The intercept's confidence interval is typically wider than the slope's because the variance of $\hat{\beta}_0$ includes an extra term involving $\bar{x}^2/S_{xx}$. When $\bar{x}$ is far from zero, the intercept estimate becomes less precise.</p> General Confidence Interval for $\beta_i$ (Multiple Regression)</h2> For any coefficient $\beta_i$ in the multiple regression model:</p> $$\hat{\beta}_i \pm t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_i),$$</p> where:</p> $$SE(\hat{\beta}_i) = \sqrt{MSE \cdot C_{ii}}$$</p> and $C_{ii}$ is the $(i+1)$-th diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</p> The degrees of freedom are $n - p$, reflecting that $p$ parameters have been estimated. As more predictors are included (larger $p$), fewer degrees of freedom remain, and the critical value $t_{\alpha/2, n-p}$ increases, widening the intervals.</p> Interpretation of Confidence Intervals</h2> A 95% confidence interval for $\beta_1$ does not mean there is a 95% probability that $\beta_1$ lies within the interval. The true parameter $\beta_1$ is a fixed (though unknown) constant; it either lies in the interval or it does not.</p> The correct interpretation is: if we were to repeat the experiment many times and construct a 95% confidence interval each time, approximately 95% of those intervals would contain the true value $\beta_1$.</p> When the Interval Contains Zero</h2> If the confidence interval for $\beta_j$ includes zero, this suggests that the data are consistent with $\beta_j = 0$, meaning the predictor $x_j$ may not have a statistically significant linear effect on $y$ at the chosen confidence level.</p> Conversely, if the interval does not include zero, we have evidence that the predictor has a significant effect. This connection between confidence intervals and hypothesis testing is explored in the next post.</p> Confidence Interval for the Mean Response</h2> We can also construct a confidence interval for the mean response at a given value $x_0$. The estimated mean response is:</p> $$\hat{y}_0 = \hat{\beta}_0 + \hat{\beta}_1 x_0.$$</p> The variance of this estimate is:</p> $$\text{Var}(\hat{y}_0) = \sigma^2\left(\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}\right).$$</p> The confidence interval for the mean response is:</p> $$\hat{y}_0 \pm t_{\alpha/2,, n-2} \cdot \sqrt{MSE\left(\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}\right)}.$$</p> Notice that this interval is narrowest when $x_0 = \bar{x}$ and grows wider as $x_0$ moves away from $\bar{x}$. This reflects the fact that we are most confident about predictions near the center of the data.</p> Summary</h2> In this post, we constructed confidence intervals for regression coefficients:</p> The pivotal quantity $(\hat{\beta}_j - \beta_j)/SE(\hat{\beta}_j) \sim t_{n-p}$ forms the basis for inference.</li> For $\beta_1$: $\hat{\beta}_1 \pm t_{\alpha/2, n-2} \cdot \sqrt{MSE/S_{xx}}$.</li> For $\beta_0$: $\hat{\beta}_0 \pm t_{\alpha/2, n-2} \cdot \sqrt{MSE(1/n + \bar{x}^2/S_{xx})}$.</li> For any $\beta_i$ in multiple regression: $\hat{\beta}_i \pm t_{\alpha/2, n-p} \cdot \sqrt{MSE \cdot C_{ii}}$.</li> Interval width depends on the confidence level, residual variability, predictor spread, and sample size.</li> </ul> In the next post, we will use the same t-distribution framework to perform formal hypothesis tests on the regression coefficients.</p> Standard Error in Regression 2026-02-06T00:00:00+00:00 This note explains standard error in regression as a measure of uncertainty in coefficient estimates, not model quality by itself. The practical question is how noisy your estimate is if the data-generating process were repeated. I connect residual variance, sample size, and confidence in coefficients.</p> Estimating the Error Variance</h2> Under the linear model $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ with $\varepsilon_i \sim N(0, \sigma^2)$, the error variance $\sigma^2$ is unknown and must be estimated from the data.</p> The natural estimate is based on the residuals. The Mean Squared Error (MSE) is defined as:</p> $$MSE = \frac{SSE}{n - 2} = \frac{\sum^n_{i=1}(y_i - \hat{y}_i)^2}{n - 2}.$$</p> The denominator is $n - 2$ (not $n$) because we lose two degrees of freedom for estimating $\hat{\beta}_0$ and $\hat{\beta}_1$. This correction ensures that MSE is an unbiased estimator of $\sigma^2$:</p> $$E(MSE) = \sigma^2.$$</p> For multiple regression with $p$ parameters, the denominator becomes $n - p$:</p> $$MSE = \frac{SSE}{n - p}.$$</p> Variance of the OLS Estimators</h2> Simple Linear Regression</h3> The OLS estimators are functions of the random variable $\mathbf{Y}$ (since $\mathbf{X}$ is treated as fixed). Their variances can be derived from the model assumptions.</p> Variance of $\hat{\beta}_1$:</strong></p> Since $\hat{\beta}_1 = S_{xy}/S_{xx} = \sum^n_{i=1}c_i y_i$ where $c_i = (x_i - \bar{x})/S_{xx}$, and the $y_i$ are independent with variance $\sigma^2$:</p> $$\text{Var}(\hat{\beta}_1) = \sum^n_{i=1}c_i^2 \text{Var}(y_i) = \sigma^2 \sum^n_{i=1}\frac{(x_i - \bar{x})^2}{S_{xx}^2} = \sigma^2 \cdot \frac{S_{xx}}{S_{xx}^2} = \frac{\sigma^2}{S_{xx}}.$$</p> Variance of $\hat{\beta}_0$:</strong></p> Since $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$:</p> $$\text{Var}(\hat{\beta}_0) = \text{Var}(\bar{y}) + \bar{x}^2 \text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{n} + \bar{x}^2 \cdot \frac{\sigma^2}{S_{xx}} = \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right).$$</p> Note that $\text{Var}(\hat{\beta}_0)$ depends on the mean of $x$: the further $\bar{x}$ is from zero, the larger the variance of the intercept estimate.</p> Standard Errors</h2> The standard error of an estimator is the square root of its estimated variance, obtained by replacing $\sigma^2$ with MSE.</p> Standard error of $\hat{\beta}_1$:</strong></p> $$SE(\hat{\beta}_1) = \sqrt{\frac{MSE}{S_{xx}}}.$$</p> Standard error of $\hat{\beta}_0$:</strong></p> $$SE(\hat{\beta}_0) = \sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right)}.$$</p> These formulas show that the precision of our estimates improves (standard errors decrease) when:</p> $MSE$ is small (the model fits the data well).</li> $S_{xx}$ is large (there is more spread in the $x$ values).</li> $n$ is large (we have more data).</li> </ul> General Case: Multiple Regression</h2> For the multiple regression model $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$, the variance-covariance matrix of $\hat{\boldsymbol{\beta}}$ is:</p> $$\text{Var}(\hat{\boldsymbol{\beta}}) = \sigma^2(\mathbf{X}^T\mathbf{X})^{-1}.$$</p> The estimated variance-covariance matrix is:</p> $$\widehat{\text{Var}}(\hat{\boldsymbol{\beta}}) = MSE \cdot (\mathbf{X}^T\mathbf{X})^{-1}.$$</p> Let $(\mathbf{X}^T\mathbf{X})^{-1} = \mathbf{C}$, and let $C_{jj}$ denote the $(j+1)$-th diagonal element of $\mathbf{C}$ (corresponding to $\hat{\beta}_j$). Then:</p> $$\text{Var}(\hat{\beta}_j) = \sigma^2 C_{jj},$$</p> $$SE(\hat{\beta}_j) = \sqrt{MSE \cdot C_{jj}}.$$</p> Verification for Simple Linear Regression</h3> For simple linear regression, the design matrix is:</p> $$\mathbf{X} = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}, \quad \mathbf{X}^T\mathbf{X} = \begin{pmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2 \end{pmatrix}.$$</p> The inverse is:</p> $$(\mathbf{X}^T\mathbf{X})^{-1} = \frac{1}{n\sum x_i^2 - (\sum x_i)^2}\begin{pmatrix} \sum x_i^2 & -\sum x_i \\ -\sum x_i & n \end{pmatrix}.$$</p> Since $n\sum x_i^2 - (\sum x_i)^2 = n \cdot S_{xx}$, the diagonal elements are:</p> $$C_{00} = \frac{\sum x_i^2}{n \cdot S_{xx}} = \frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}, \quad C_{11} = \frac{n}{n \cdot S_{xx}} = \frac{1}{S_{xx}}.$$</p> These give us:</p> $$\text{Var}(\hat{\beta}_0) = \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right), \quad \text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{S_{xx}},$$</p> which matches our earlier derivations.</p> Interpreting the Standard Error</h2> The standard error measures the typical size of the sampling variability in the estimated coefficient. If we were to repeatedly draw new samples of size $n$ from the same population and compute $\hat{\beta}_1$ each time, the standard deviation of these estimates would be approximately $SE(\hat{\beta}_1)$.</p> A small standard error relative to the coefficient estimate suggests that the estimate is precise. A large standard error suggests substantial uncertainty. We will use the standard error to construct confidence intervals (Post 7) and perform hypothesis tests (Post 8).</p> Summary</h2> In this post, we derived the standard errors of the regression coefficients:</p> $MSE = SSE/(n-2)$ estimates the error variance $\sigma^2$ for simple linear regression, or $MSE = SSE/(n-p)$ for multiple regression.</li> $SE(\hat{\beta}_1) = \sqrt{MSE/S_{xx}}$ measures the precision of the slope estimate.</li> $SE(\hat{\beta}_0) = \sqrt{MSE(1/n + \bar{x}^2/S_{xx})}$ measures the precision of the intercept estimate.</li> For multiple regression, $SE(\hat{\beta}_j) = \sqrt{MSE \cdot C_{jj}}$ where $C_{jj}$ is the $(j+1)$-th diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</li> Precision improves with larger $S_{xx}$, smaller $MSE$, and larger $n$.</li> </ul> In the next post, we will use these standard errors to construct confidence intervals for $\beta_0$, $\beta_1$, and $\beta_i$.</p> R-squared: SST, SSE, SSR and the Relationship with Correlation 2026-01-30T00:00:00+00:00 This note explains SST, SSR, SSE, and R-squared, then connects these quantities to correlation in the simple regression setting. The aim is to clarify what variance decomposition actually tells you about model fit. It also points out where high R-squared can still be misleading.</p> In the previous post</a>, we introduced the Pearson correlation coefficient $r$. In this post, we define the coefficient of determination $R^2$ through the decomposition of total variability, and then prove that $R^2 = r^2$ for simple linear regression.</p> The Sum of Squares Decomposition</h2> The total variability in the observed response values can be decomposed into two components: the variability explained by the regression model and the variability left unexplained (the residuals).</p> Total Sum of Squares (SST)</h3> $$SST = \sum^n_{i=1}(y_i - \bar{y})^2 = S_{yy}.$$</p> This measures the total variability of $y$ around its mean.</p> Sum of Squares due to Regression (SSR)</h3> $$SSR = \sum^n_{i=1}(\hat{y}_i - \bar{y})^2.$$</p> This measures the variability in $y$ that is explained by the regression model.</p> Sum of Squared Errors (SSE)</h3> $$SSE = \sum^n_{i=1}(y_i - \hat{y}_i)^2.$$</p> This measures the variability in $y$ that remains unexplained.</p> The Identity</h3> The fundamental decomposition is:</p> $$SST = SSR + SSE.$$</p> To see why this holds, write:</p> $$y_i - \bar{y} = (\hat{y}_i - \bar{y}) + (y_i - \hat{y}_i).$$</p> Squaring both sides and summing over all observations:</p> $$\sum_{i=1}^n (y_i - \bar{y})^2 = \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i=1}^n (y_i - \hat{y}_i)^2 + 2\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i).$$</p> The cross term vanishes because of the properties of OLS residuals:</p> $$\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) = \sum_{i=1}^n (\hat{y}_i - \bar{y}),e_i = 0.$$</p> This holds because OLS residuals are orthogonal to the fitted values. Therefore:</p> $$SST = SSR + SSE.$$</p> The Coefficient of Determination $R^2$</h2> The coefficient of determination is defined as:</p> $$R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}.$$</p> Both expressions are equivalent because $SSR = SST - SSE$.</p> The value of $R^2$ represents the proportion of the total variability in $y$ that is explained by the regression model:</p> $R^2 = 1$: the model explains all the variability (all data points lie exactly on the fitted line).</li> $R^2 = 0$: the model explains none of the variability (the fitted line is just $\hat{y} = \bar{y}$).</li> $0 \leq R^2 \leq 1$: in general, $R^2$ lies between 0 and 1 for models with an intercept.</li> </ul> Proof that $R^2 = r^2$ for Simple Linear Regression</h2> We now prove that for simple linear regression, the coefficient of determination equals the square of the Pearson correlation coefficient.</p> Step 1: Express SSR in terms of $S_{xx}$, $S_{xy}$</h3> For simple linear regression, the fitted values are:</p> $$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i = (\bar{y} - \hat{\beta}_1\bar{x}) + \hat{\beta}_1 x_i = \bar{y} + \hat{\beta}_1(x_i - \bar{x}).$$</p> Therefore:</p> $$\hat{y}_i - \bar{y} = \hat{\beta}_1(x_i - \bar{x}).$$</p> Substituting into SSR:</p> $$SSR = \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 = \hat{\beta}_1^2 \sum_{i=1}^n (x_i - \bar{x})^2 = \hat{\beta}_1^2 \cdot S_{xx}.$$</p> Since $\hat{\beta}_1 = S_{xy}/S_{xx}$:</p> $$SSR = \left(\frac{S_{xy}}{S_{xx}}\right)^2 S_{xx} = \frac{S_{xy}^2}{S_{xx}}.$$</p> Step 2: Compute $R^2$</h3> $$R^2 = \frac{SSR}{SST} = \frac{S_{xy}^2 / S_{xx}}{S_{yy}} = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}}.$$</p> Step 3: Compare with $r^2$</h3> From the previous post</a>, the correlation coefficient is:</p> $$r = \frac{S_{xy}}{\sqrt{S_{xx} \cdot S_{yy}}}.$$</p> Squaring both sides:</p> $$r^2 = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}}.$$</p> Therefore:</p> $$R^2 = r^2. \quad \blacksquare$$</p> This result is elegant and important. It tells us that for simple linear regression, the proportion of variance explained by the model is exactly the square of the correlation between $x$ and $y$.</p> Alternative Proof via SSE</h2> We can also prove the result by expressing SSE in terms of $S_{xx}$, $S_{yy}$, $S_{xy}$.</p> Since $e_i = y_i - \hat{y}_i = y_i - \bar{y} - \hat{\beta}_1(x_i - \bar{x})$:</p> $$\begin{aligned} SSE &= \sum_{i=1}^n \big[(y_i - \bar{y}) - \hat{\beta}_1 (x_i - \bar{x})\big]^2 \\ &= \sum_{i=1}^n (y_i - \bar{y})^2 - 2\hat{\beta}_1 \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) + \hat{\beta}_1^2 \sum_{i=1}^n (x_i - \bar{x})^2 \\ &= S_{yy} - 2\hat{\beta}_1 S_{xy} + \hat{\beta}_1^2 S_{xx}. \end{aligned}$$</p> Substituting $\hat{\beta}_1 = S_{xy}/S_{xx}$:</p> $$\begin{aligned} SSE &= S_{yy} - 2\cdot\frac{S_{xy}}{S_{xx}}\cdot S_{xy} + \frac{S_{xy}^2}{S_{xx}^2}\cdot S_{xx} \\ &= S_{yy} - \frac{2S_{xy}^2}{S_{xx}} + \frac{S_{xy}^2}{S_{xx}} \\ &= S_{yy} - \frac{S_{xy}^2}{S_{xx}}. \end{aligned}$$</p> Therefore:</p> $$R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{S_{yy} - S_{xy}^2/S_{xx}}{S_{yy}} = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}} = r^2.$$</p> Adjusted $R^2$ for Multiple Regression</h2> For multiple regression with $p$ parameters, $R^2$ always increases (or stays the same) when additional predictors are added, even if those predictors have no real relationship with $y$. To account for this, the adjusted $R^2$ penalizes for the number of predictors:</p> $$R^2_{\text{adj}} = 1 - \frac{SSE/(n - p)}{SST/(n - 1)} = 1 - \frac{n - 1}{n - p}(1 - R^2).$$</p> Unlike $R^2$, the adjusted version can decrease when an uninformative predictor is added, making it more suitable for model comparison.</p> Note that for simple linear regression ($p = 2$), $R^2 = r^2$ still holds, and $R^2_{\text{adj}}$ simplifies accordingly.</p> Where a High R-squared Can Mislead</h2> Everything above describes what $R^2$ measures. It is equally important to be clear about what it does not measure.</p> First, $R^2$ quantifies fit, not correctness. It tells us how much of the variability in $y$ is captured by the fitted line, but it says nothing about whether a line was the right model to fit. A clearly nonlinear pattern can still produce a high $R^2$ if the linear component happens to be strong. Anscombe's quartet is the classic reference here: four datasets with essentially identical $R^2$ (and identical fitted lines), one of which is a clean linear relationship, one a perfect parabola, and two dominated by single outlying points. The residual plot distinguishes them; $R^2$ does not.</p> Second, plain $R^2$ never decreases when a predictor is added. Since OLS minimizes SSE over a strictly larger space of models, SSE can only stay the same or shrink, so $R^2 = 1 - SSE/SST$ can only stay the same or grow, even when the new predictor is pure noise. This is exactly the problem the adjusted $R^2$ above is designed to correct.</p> Third, a high $R^2$ is not evidence of causation, and it is not a measure of predictive performance on new data. It is computed on the same observations used to fit the model, so it says nothing about how the model generalizes out of sample.</p> Summary</h2> In this post, we covered the sum of squares decomposition and the coefficient of determination:</p> $SST = SSR + SSE$ decomposes total variability into explained and unexplained components.</li> $R^2 = SSR/SST = 1 - SSE/SST$ measures the proportion of variance explained.</li> For simple linear regression, $R^2 = r^2 = S_{xy}^2/(S_{xx} \cdot S_{yy})$, proven by expressing SSR in terms of $S_{xx}$ and $S_{xy}$.</li> Adjusted $R^2$ penalizes for model complexity in multiple regression.</li> </ul> In the next post, we will derive the standard errors of the regression coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$ using $S_{xx}$ and MSE.</p> Correlation 2026-01-23T00:00:00+00:00 This note defines correlation clearly, explains how to compute it, and separates association from causation. The key issue is that correlation is often over-interpreted as evidence of mechanism. The goal is to make correlation a diagnostic tool, not a conclusion.</p> In the previous posts, we built the regression framework using $S_{xx}$ and $S_{xy}$ to derive the OLS estimators. Before we move to deeper inferential topics, it is essential to formalize a measure of the strength of the linear association between two variables. This measure is the Pearson correlation coefficient.</p> Recap of Key Notation</h2> In Simple Linear Regression</a>, we introduced:</p> $$S_{xx} = \sum^n_{i=1}(x_i - \bar{x})^2,$$</p> $$S_{xy} = \sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y}).$$</p> We now introduce the remaining quantity:</p> $$S_{yy} = \sum^n_{i=1}(y_i - \bar{y})^2.$$</p> These three summary statistics capture the variability of $x$, the variability of $y$, and their joint variability, respectively.</p> Sample Standard Deviations</h2> The sample standard deviations are defined as:</p> $$S_x = \sqrt{\frac{S_{xx}}{n - 1}} = \sqrt{\frac{1}{n-1}\sum^n_{i=1}(x_i - \bar{x})^2},$$</p> $$S_y = \sqrt{\frac{S_{yy}}{n - 1}} = \sqrt{\frac{1}{n-1}\sum^n_{i=1}(y_i - \bar{y})^2}.$$</p> Note that $S_x$ and $S_y$ are the standard deviations using the $n - 1$ denominator (Bessel's correction), which gives unbiased estimates of the population standard deviations $\sigma_x$ and $\sigma_y$.</p> The sample variances are simply:</p> $$S_x^2 = \frac{S_{xx}}{n-1}, \quad S_y^2 = \frac{S_{yy}}{n-1}.$$</p> The Pearson Correlation Coefficient</h2> The sample Pearson correlation coefficient is defined as:</p> $$r = \frac{S_{xy}}{\sqrt{S_{xx} \cdot S_{yy}}}.$$</p> Equivalently, using the standard deviations:</p> $$r = \frac{\sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y})}{(n-1) S_x S_y} = \frac{S_{xy}}{(n-1) S_x S_y}.$$</p> The first form (using $S_{xx}$, $S_{yy}$, $S_{xy}$) is often more convenient for algebraic manipulation, while the second form highlights that $r$ is a standardized measure of covariation.</p> Properties of the Correlation Coefficient</h2> The Pearson correlation coefficient has several important properties:</p> 1. Bounded between -1 and 1</h3> $$-1 \leq r \leq 1.$$</p> This follows from the Cauchy-Schwarz inequality, which states that $(S_{xy})^2 \leq S_{xx} \cdot S_{yy}$, with equality only when all data points lie exactly on a line.</p> 2. Interpretation of values</h3> $r = 1$: perfect positive linear relationship (all points on a line with positive slope).</li> $r = -1$: perfect negative linear relationship (all points on a line with negative slope).</li> $r = 0$: no linear relationship (but there may be a nonlinear relationship).</li> $0 < \|r\| < 1$: partial linear association, with strength increasing as $\|r\|$ approaches 1.</li> </ul> 3. Symmetry</h3> $$r_{xy} = r_{yx}.$$</p> The correlation between $x$ and $y$ is the same as the correlation between $y$ and $x$. This follows because $S_{xy}$ is symmetric in $x$ and $y$.</p> 4. Invariance under linear transformation</h3> If we define $u_i = a + bx_i$ and $v_i = c + dy_i$ with $b > 0$ and $d > 0$, then the correlation between $u$ and $v$ equals the correlation between $x$ and $y$. If either $b$ or $d$ is negative (but not both), the sign of $r$ flips.</p> 5. Dimensionless</h3> The correlation coefficient has no units. The numerator $S_{xy}$ has units of $x$ times $y$, and the denominator $\sqrt{S_{xx} \cdot S_{yy}}$ also has units of $x$ times $y$, so they cancel.</p> Relationship to the Regression Slope</h2> Recall from Post 1 that the OLS slope estimator is $\hat{\beta}_1 = S_{xy}/S_{xx}$. The correlation coefficient can be expressed in terms of $\hat{\beta}_1$:</p> $$r = \hat{\beta}_1 \sqrt{\frac{S_{xx}}{S_{yy}}} = \hat{\beta}_1 \cdot \frac{S_x}{S_y}.$$</p> This shows that $r$ and $\hat{\beta}_1$ always share the same sign. A positive slope corresponds to a positive correlation, and a negative slope corresponds to a negative correlation.</p> Conversely, we can write the slope as:</p> $$\hat{\beta}_1 = r \cdot \frac{S_y}{S_x}.$$</p> This expresses the regression slope as the correlation times the ratio of the standard deviations.</p> Population Correlation</h2> The sample correlation $r$ estimates the population correlation coefficient $\rho$ (rho), defined for a bivariate population as:</p> $$\rho = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y} = \frac{E[(X - \mu_X)(Y - \mu_Y)]}{\sigma_X \sigma_Y}.$$</p> When the data $(x_i, y_i)$ are sampled from a bivariate normal distribution, $r$ is a consistent estimator of $\rho$.</p> Correlation Does Not Imply Causation</h2> A strong correlation between two variables does not mean that one causes the other. The association might be due to a lurking variable, reverse causation, or coincidence. Regression models describe associations; establishing causation requires careful experimental design or additional assumptions.</p> Summary</h2> In this post, we introduced the Pearson correlation coefficient:</p> $S_{yy} = \sum(y_i - \bar{y})^2$ completes the set of summary statistics alongside $S_{xx}$ and $S_{xy}$.</li> The sample standard deviations $S_x = \sqrt{S_{xx}/(n-1)}$ and $S_y = \sqrt{S_{yy}/(n-1)}$ measure spread.</li> The correlation $r = S_{xy}/\sqrt{S_{xx} \cdot S_{yy}}$ quantifies the strength of the linear relationship.</li> $r$ is bounded between $-1$ and $1$, symmetric, and dimensionless.</li> The regression slope and correlation are related by $\hat{\beta}_1 = r \cdot S_y / S_x$.</li> </ul> In the next post, we will define $R^2$ using SST, SSE, and SSR, and prove mathematically that $R^2 = r^2$ for simple linear regression.</p> Polynomial Regression 2026-01-16T00:00:00+00:00 This note explains polynomial regression as a linear model in transformed features, not a different estimation framework. The motivation is to capture nonlinear trends while retaining familiar regression machinery. I highlight when polynomial terms help and when they mainly increase variance.</p> In the previous post</a>, we introduced multiple linear regression using matrix notation. One natural question arises: what if the relationship between $x$ and $y$ is not a straight line? Polynomial regression addresses this by fitting a polynomial function of a single variable $x$, and it turns out to be a special case of the multiple linear regression framework we have already developed.</p> The Polynomial Model</h2> A polynomial regression model of degree $d$ takes the form:</p> $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_d x_i^d + \varepsilon_i.$$</p> This model is nonlinear in the variable $x$, but it is still linear in the parameters $\beta_0, \beta_1, \ldots, \beta_d$. This distinction is important because it means we can use the same OLS estimation procedure from multiple linear regression.</p> For example, a quadratic model ($d = 2$) is:</p> $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \varepsilon_i,$$</p> and a cubic model ($d = 3$) is:</p> $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \varepsilon_i.$$</p> Design Matrix Construction</h2> To fit a polynomial regression using the matrix approach, we define new variables:</p> $$z_1 = x, \quad z_2 = x^2, \quad z_3 = x^3, \quad \ldots, \quad z_d = x^d.$$</p> The design matrix for a polynomial of degree $d$ with $n$ observations is:</p> $$\mathbf{X} = \begin{pmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^d \\ 1 & x_2 & x_2^2 & \cdots & x_2^d \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^d \end{pmatrix}.$$</p> This is exactly the design matrix for a multiple linear regression with $d$ predictors $z_1, z_2, \ldots, z_d$. Therefore, the OLS estimator is:</p> $$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y},$$</p> which is the same formula we derived in the previous post.</p> Why Polynomial Regression is a Special Case of MLR</h2> The key insight is that "linear" in "linear regression" refers to linearity in the parameters, not in the predictors. Although the polynomial model includes terms like $x^2$ and $x^3$, each coefficient $\beta_j$ appears linearly. We can treat each power of $x$ as a separate predictor variable, and the entire OLS theory from multiple linear regression applies directly.</p> This means all the results we derived earlier carry over:</p> The normal equations $\mathbf{X}^T\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{Y}$ hold.</li> The hat matrix $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ projects $\mathbf{Y}$ onto $\hat{\mathbf{Y}}$.</li> Residual properties remain the same.</li> </ul> Choosing the Degree</h2> A natural question is: what degree $d$ should we use? The choice involves a tradeoff between model flexibility and model complexity.</p> Too low a degree</strong> (underfitting): The model is not flexible enough to capture the true relationship, leading to large systematic errors.</li> Too high a degree</strong> (overfitting): The model fits the training data very closely, including the noise, but performs poorly on new data.</li> </ul> A polynomial of degree $n - 1$ (where $n$ is the number of data points) can pass through every data point exactly, giving $SSE = 0$. However, such a model almost always overfits and generalizes badly.</p> Common approaches to select the degree include:</p> Visual inspection</strong>: Plot the data and the fitted curve for different degrees, and choose the one that captures the trend without fitting the noise.</li> Cross-validation</strong>: Split the data into training and validation sets, fit models of various degrees on the training set, and select the degree with the lowest prediction error on the validation set.</li> Information criteria</strong>: Use metrics such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) to balance goodness of fit with model complexity.</li> </ol> Multicollinearity Considerations</h2> One practical concern with polynomial regression is multicollinearity. The predictors $x, x^2, x^3, \ldots$ are often highly correlated, especially when $x$ values span a narrow range. High multicollinearity inflates the variance of the coefficient estimates and makes $\mathbf{X}^T\mathbf{X}$ nearly singular.</p> A common remedy is to center the predictor before constructing polynomial terms. Instead of using $x$, we use $x - \bar{x}$:</p> $$z_1 = x - \bar{x}, \quad z_2 = (x - \bar{x})^2, \quad z_3 = (x - \bar{x})^3, \quad \ldots$$</p> Centering reduces the correlation among the polynomial terms and improves the numerical stability of the estimation.</p> Another approach is to use orthogonal polynomials, which are constructed so that $\mathbf{X}^T\mathbf{X}$ is diagonal, eliminating multicollinearity entirely.</p> Example: Quadratic Fit</h2> Consider a quadratic model $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \varepsilon_i$ with the design matrix:</p> $$\mathbf{X} = \begin{pmatrix} 1 & x_1 & x_1^2 \\ 1 & x_2 & x_2^2 \\ \vdots & \vdots & \vdots \\ 1 & x_n & x_n^2 \end{pmatrix}.$$</p> There are $p = 3$ parameters. The OLS solution $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ gives us $\hat{\beta}_0$, $\hat{\beta}_1$, and $\hat{\beta}_2$ simultaneously. The coefficient $\hat{\beta}_2$ indicates the curvature of the fitted parabola: a positive $\hat{\beta}_2$ means the curve opens upward, and a negative $\hat{\beta}_2$ means it opens downward.</p> Summary</h2> In this post, we explored polynomial regression:</p> The polynomial model $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_d x_i^d + \varepsilon_i$ is nonlinear in $x$ but linear in the parameters.</li> By treating each power of $x$ as a separate predictor, polynomial regression becomes a special case of multiple linear regression.</li> The OLS estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ applies directly.</li> Choosing the polynomial degree requires balancing fit and complexity to avoid overfitting.</li> Centering or using orthogonal polynomials helps address multicollinearity.</li> </ul> In the next post, we will examine correlation, which quantifies the strength of the linear relationship between two variables using $S_{xx}$, $S_{yy}$, and $S_{xy}$.</p> Multiple Linear Regression 2026-01-09T00:00:00+00:00 This note extends linear regression to multiple predictors and explains how interpretation changes once features interact through a shared model. The practical challenge is understanding coefficients conditionally, not in isolation. I focus on the model form, estimation intuition, and common interpretation mistakes.</p> In the previous post</a>, we explored simple linear regression with a single predictor. In practice, the response variable $y$ often depends on more than one predictor. Multiple linear regression extends the framework to accommodate $p - 1$ independent variables.</p> The Model</h2> The multiple linear regression model for the $i$-th observation is:</p> $$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_{p-1} x_{i,p-1} + \varepsilon_i,$$</p> where $x_{ij}$ is the value of the $j$-th predictor for the $i$-th observation, $\beta_j$ are the regression coefficients, and $\varepsilon_i$ is the error term. The total number of parameters is $p$ (including the intercept $\beta_0$).</p> Matrix Notation</h2> Writing out the model for each observation individually becomes cumbersome as the number of predictors grows. Matrix notation provides a compact and powerful alternative.</p> Define the following:</p> $$\mathbf{Y} = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}, \quad \mathbf{X} = \begin{pmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1,p-1} \\ 1 & x_{21} & x_{22} & \cdots & x_{2,p-1} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{n,p-1} \end{pmatrix},$$</p> $$\boldsymbol{\beta} = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_{p-1} \end{pmatrix}, \quad \boldsymbol{\varepsilon} = \begin{pmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{pmatrix}.$$</p> The model can now be written as:</p> $$\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}.$$</p> Here, $\mathbf{Y}$ is an $n \times 1$ vector of responses, $\mathbf{X}$ is an $n \times p$ design matrix (the first column of ones accounts for the intercept), $\boldsymbol{\beta}$ is a $p \times 1$ vector of coefficients, and $\boldsymbol{\varepsilon}$ is an $n \times 1$ vector of errors.</p> OLS in Matrix Form</h2> The sum of squared errors can be written in matrix form as:</p> $$SSE = (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})^T(\mathbf{Y} - \mathbf{X}\boldsymbol{\beta}).$$</p> Expanding this expression:</p> $$SSE = \mathbf{Y}^T\mathbf{Y} - 2\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{Y} + \boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}.$$</p> To minimize, we take the derivative with respect to $\boldsymbol{\beta}$ and set it to zero:</p> $$\frac{\partial SSE}{\partial \boldsymbol{\beta}} = -2\mathbf{X}^T\mathbf{Y} + 2\mathbf{X}^T\mathbf{X}\boldsymbol{\beta} = \mathbf{0}.$$</p> This gives us the normal equations in matrix form:</p> $$\mathbf{X}^T\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{Y}.$$</p> Provided that $\mathbf{X}^T\mathbf{X}$ is invertible (that is, the columns of $\mathbf{X}$ are linearly independent), we can solve for the OLS estimator:</p> $$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}.$$</p> This single formula generalizes the simple linear regression result. When $p = 2$ (one predictor plus the intercept), this reduces to $\hat{\beta}1 = S</em>{xy}/S_{xx}$ and $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$ as derived in the previous post.</p> Interpreting the Coefficients</h2> Each coefficient $\hat{\beta}_j$ (for $j = 1, 2, \ldots, p - 1$) represents the estimated change in $y$ for a one-unit increase in $x_j$, while holding all other predictors constant. This "holding other variables constant" interpretation is what distinguishes multiple regression from running separate simple regressions.</p> The intercept $\hat{\beta}_0$ represents the estimated value of $y$ when all predictors are equal to zero.</p> The Hat Matrix</h2> The vector of fitted values is:</p> $$\hat{\mathbf{Y}} = \mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y} = \mathbf{H}\mathbf{Y},$$</p> where $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ is called the hat matrix. It "puts a hat on" $\mathbf{Y}$, transforming observed values into fitted values. The hat matrix is symmetric ($\mathbf{H}^T = \mathbf{H}$) and idempotent ($\mathbf{H}^2 = \mathbf{H}$).</p> The residual vector is:</p> $$\mathbf{e} = \mathbf{Y} - \hat{\mathbf{Y}} = (\mathbf{I} - \mathbf{H})\mathbf{Y}.$$</p> Assumptions</h2> The assumptions for multiple linear regression extend those of simple linear regression:</p> Linearity</strong>: $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$, the model is linear in the parameters.</li> Full rank</strong>: The design matrix $\mathbf{X}$ has full column rank, meaning $\text{rank}(\mathbf{X}) = p$. This ensures $\mathbf{X}^T\mathbf{X}$ is invertible.</li> Exogeneity</strong>: $E(\boldsymbol{\varepsilon} \| \mathbf{X}) = \mathbf{0}$, the errors have zero conditional mean.</li> Homoscedasticity</strong>: $\text{Var}(\boldsymbol{\varepsilon} \| \mathbf{X}) = \sigma^2\mathbf{I}_n$, the errors have constant variance and are uncorrelated.</li> Normality</strong> (for inference): $\boldsymbol{\varepsilon} \sim N(\mathbf{0}, \sigma^2\mathbf{I}_n)$.</li> </ol> When assumptions 1 through 4 hold, the Gauss-Markov theorem guarantees that $\hat{\boldsymbol{\beta}}$ is the Best Linear Unbiased Estimator (BLUE).</p> Connection to Simple Linear Regression</h2> In the special case where $p = 2$, we have a single predictor $x$ and the design matrix becomes:</p> $$\mathbf{X} = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}.$$</p> Computing $(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ in this case yields the familiar results $\hat{\beta}1 = S</em>{xy}/S_{xx}$ and $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$, confirming that the matrix formulation is a true generalization.</p> Summary</h2> In this post, we extended the regression framework to handle multiple predictors:</p> The model $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$ uses matrix notation to express the relationship compactly.</li> The OLS estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ generalizes the simple linear regression solution.</li> Each $\hat{\beta}_j$ measures the effect of one predictor while holding the others constant.</li> The hat matrix $\mathbf{H}$ projects observed values onto fitted values.</li> </ul> In the next post, we will explore polynomial regression, which is a special case of multiple linear regression where the predictors are powers of a single variable.</p> Simple Linear Regression 2026-01-02T00:00:00+00:00 This note introduces simple linear regression from first principles and focuses on how slope and intercept are estimated from data. The problem it solves is modeling a linear relationship between one predictor and one response. It is intended as the base layer for the later regression notes in this series.</p> In secondary school, we learn that the equation of a straight line is given by $y = mx + c$, where $m$ is the slope and $c$ is the y-intercept. In statistics and machine learning, we use a similar but more general form to model the relationship between a dependent variable $y$ and an independent variable $x$. This is known as simple linear regression.</p> The Model</h2> In simple linear regression, we express the relationship as:</p> $$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i,$$</p> where $\hat{y}_i$ is the predicted value of the dependent variable for the $i$-th observation, $\hat{\beta}_0$ is the estimated y-intercept, and $\hat{\beta}_1$ is the estimated slope coefficient. The "hat" notation indicates that these are estimates derived from data, not the true (unknown) population parameters $\beta_0$ and $\beta_1$.</p> The true model is assumed to be:</p> $$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i,$$</p> where $\varepsilon_i$ represents the random error term for the $i$-th observation.</p> Ordinary Least Squares (OLS)</h2> The question is: how do we find the best estimates $\hat{\beta}_0$ and $\hat{\beta}_1$? We need a systematic method that determines the line of best fit. The most common approach is Ordinary Least Squares (OLS).</p> OLS minimizes the sum of the squared differences between the observed values $y_i$ and the predicted values $\hat{y}_i$. These differences are called residuals, defined as $e_i = y_i - \hat{y}_i$. By squaring the residuals, we treat positive and negative errors equally and penalize larger deviations more heavily.</p> The objective function is the Sum of Squared Errors (SSE):</p> $$SSE = \sum^n_{i=1}(y_i - \hat{y}_i)^2 = \sum^n_{i=1}(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2.$$</p> Deriving the Normal Equations</h2> To minimize the SSE, we take partial derivatives with respect to $\hat{\beta}_0$ and $\hat{\beta}_1$ and set them equal to zero.</p> Partial derivative with respect to $\hat{\beta}_0$</h3> $$\frac{\partial SSE}{\partial \hat{\beta}_0} = -2\sum^n_{i=1}(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0.$$</p> Dividing both sides by $-2$ and expanding the sum:</p> $$\sum^n_{i=1} y_i - n\hat{\beta}_0 - \hat{\beta}_1 \sum^n_{i=1} x_i = 0.$$</p> Solving for $\hat{\beta}_0$:</p> $$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x},$$</p> where $\bar{x} = \frac{1}{n}\sum^n_{i=1}x_i$ and $\bar{y} = \frac{1}{n}\sum^n_{i=1}y_i$ are the sample means.</p> Partial derivative with respect to $\hat{\beta}_1$</h3> $$\frac{\partial SSE}{\partial \hat{\beta}_1} = -2\sum^n_{i=1}x_i(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0.$$</p> Dividing by $-2$ and expanding:</p> $$\sum^n_{i=1} x_i y_i - \hat{\beta}_0 \sum^n_{i=1} x_i - \hat{\beta}_1 \sum^n_{i=1} x_i^2 = 0.$$</p> Substituting $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$:</p> $$\sum^n_{i=1} x_i y_i - (\bar{y} - \hat{\beta}_1 \bar{x})\sum^n_{i=1} x_i - \hat{\beta}_1 \sum^n_{i=1} x_i^2 = 0.$$</p> After simplification, we obtain:</p> $$\hat{\beta}_1 = \frac{\sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y})}{\sum^n_{i=1}(x_i - \bar{x})^2}.$$</p> Introducing $S_{xx}$ and $S_{xy}$ Notation</h2> To write the estimators more concisely, we define the following summary statistics:</p> $$S_{xx} = \sum^n_{i=1}(x_i - \bar{x})^2 = \sum^n_{i=1}x_i^2 - n\bar{x}^2,$$</p> $$S_{xy} = \sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y}) = \sum^n_{i=1}x_i y_i - n\bar{x}\bar{y}.$$</p> Using this notation, the OLS estimators become:</p> $$\hat{\beta}_1 = \frac{S_{xy}}{S_{xx}},$$</p> $$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}.$$</p> These are elegant expressions that reveal the structure of the estimates. The slope $\hat{\beta}_1$ is the ratio of the joint variability of $x$ and $y$ (captured by $S_{xy}$) to the variability of $x$ alone (captured by $S_{xx}$). The intercept $\hat{\beta}_0$ ensures the regression line passes through the point $(\bar{x}, \bar{y})$.</p> Residuals and Fitted Values</h2> Once we have $\hat{\beta}_0$ and $\hat{\beta}_1$, we can compute:</p> Fitted values</strong>: $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ for each observation.</li> Residuals</strong>: $e_i = y_i - \hat{y}_i$ for each observation.</li> </ul> Two important properties of OLS residuals are worth noting:</p> The residuals sum to zero: $\sum^n_{i=1} e_i = 0$.</li> The residuals are uncorrelated with the fitted values: $\sum^n_{i=1} e_i \hat{y}_i = 0$.</li> </ol> These properties follow directly from the normal equations.</p> Key Assumptions</h2> For OLS to produce reliable estimates, the following assumptions are typically required:</p> Linearity</strong>: The relationship between $x$ and $y$ is linear in the parameters.</li> Independence</strong>: The observations are independent of one another.</li> Homoscedasticity</strong>: The variance of the error terms $\varepsilon_i$ is constant across all values of $x$.</li> Normality</strong>: The error terms are normally distributed, that is, $\varepsilon_i \sim N(0, \sigma^2)$.</li> </ol> When these assumptions hold, OLS produces the Best Linear Unbiased Estimators (BLUE) according to the Gauss-Markov theorem.</p> Summary</h2> In this post, we covered the fundamentals of simple linear regression:</p> The model $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ describes the estimated linear relationship between $x$ and $y$.</li> OLS minimizes the sum of squared errors to find the best-fitting line.</li> The estimators $\hat{\beta}1 = S</em>{xy}/S_{xx}$ and $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$ are derived from the normal equations.</li> The $S_{xx}$ and $S_{xy}$ notation provides a compact way to express these results.</li> </ul> In the next post, we will extend this framework to handle multiple independent variables through multiple linear regression.</p> Let's talk about Solar Photovoltaic Systems in WWTPs Malaysia 2025-12-01T00:00:00+00:00 This note reviews how solar photovoltaic systems can reduce energy cost and emissions in wastewater treatment plants (WWTPs) in Malaysia. The core problem is that WWTP operations are electricity-intensive and exposed to tariff volatility. I summarize constraints, control architecture, and where ML-based forecasting can improve deployment decisions.</p> Why Solar PV in WWTPs?</h1> Wastewater treatment facilities are globally recognized as energy-intensive operations, often relying on electrical energy accounting for a significant portion of their operational expenditure (OPEX). For conventional WWTPs, energy use contributes between 25%-60% of the total OPEX ^{Muzaffar2022</sup> . This high dependency on the central electricity grid exposes these facilities to electricity tariffs volatility and supply uncertainties. The national sewerage company, Indah Water Konsortium (IWK) Sdn Bhd, has experienced a substantial increase with electricity costs ballooned from RM22.53 million in 2000 to RM256.30 million in 2020 ^{IWK2021</sup> . This became the primary motivation for IWK to explore renewable energy sources, particularly solar PV systems, to mitigate energy costs and enhance sustainability.</p>}} IWK operates as Malaysia's national sewerage company and manages a vast network of public treatment plants. As of December 2021, IWK operated and maintained 7,272 public Sewerage Treatment Plants (STPs) and 1,375 network pump stations across the country ^{IWK2021_report</sup> . The regulatory oversight is provided by the Suruhanjaya Perkhidmatan Air Negara (SPAN), which ensures compliance with national water quality, ensuring that operators comply with stipulated standards and contractual obligations.</p>} Energy Demand Characteristics of WWTPs</h1> Before implementing solar PV systems, it is crucial to understand the energy demand characteristics of WWTPs to correctly sizing and optimizing a solar PV system. The energy consumption profile of WWTP is highly influence by the pkant scale and the dominant treatment technology used.</p> Specific Energy Consumption (SEC) and Categorization</h2> Energy intensity, generally measured as Specific Energy Consumption (SEC) in kilowatt-hour per cubic meter (kWh/m³) of treated wastewater, is a key metric for evaluating the energy efficiency of WWTPs. The SEC values can vary significantly based on the treatment processes employed and the plant's capacity. Data suggests that smaller WWTPs, particular those below $10,000 m^3/month$ capacity, tends to bear a disproportionately higher electricity cost, accounting for 30%-40% of their total running costs ^{Muzaffar2022</sup> . In contrast, larger WWTPs with capacities exceeding typically exhibit lower SEC values, accounting for 15%-30% of the total running costs.</p>} Specific data collected from village WWTPs in Romania indicated high annual average SEC values, ranging between $1.786 kWh/m^3$ to $2.334 kWh/m^3$ of treated wastewater ^{Tokos2021</sup> . Furthermore, the complexity of sewage sludge disposal capacity has increased rapidly alongside population growthm reaching 7 millions $m^3$ annually. Managing this sludge requires significant energy input, contributing to the sector's overall energy consumption, with sewage sludge treatment alone consuming an estimated $544,900 GWh$ across IWK operations between 2016 and 2019 ^{Quan2022</sup> .</p>}} This observed variability indicates that the primary candidates for immediate solar PV intergartion are the small to medium sized conventional STPs. These plants have higher SEC values and higher dependence on expensive grid power, meaninf the marginal return from solar PV energy offset is maximized in this segment.</p> Identification of Energy-Intensive Processes</h2> The greatest energy concentration of electrical energy demand within a conventional WWTP is typically found in the biological treatment line (BgT), primarily driven by the aeration systems. This process is necessary for activated sludge processes to maintain adequate dissolved oxygen levels for microbial activity. Data confirms that BgT accounts for the majority of total energy consumption, ranging from 63.2% to 72.9% in surveyed WWTPs ^{Tokos2021</sup> .</p>}Whether thorugh diffused air of mechanical aerators (such as those used in oxidation ditches</a>), aeration systems requires continuous operation to provide oxygen transfer, circulation, and mixing. This need for continuous, high-power during daytime operational hours, and we realized this characteristic is highly advantageous for solar PV integration. The steady daytime demand for blowers and aerators means that the generated solar power can be immediately and entirely consumed, leadning to high utilization rates, and diminish the need for Battery Energy Storage Systems (BESS), which is often a significant cost driver in renewable energy (RE) projects.</p> The high concentration of energy use in aeration suggests that before implementing solar PV systems, WWTP operators should measures energy efficiency in the aeration systems. Simply installing a large PV array without optimzing the blowers and biological processes may lead to a larger and more costly system. Streamlining the anaerobic biological treatments is possible and should be prioritized to reduce the overall energy demand before sizing the solar PV system.</p> Advanced PV Power Forecasting and Control Systems</h1> The modern integration of photovoltaic systems into WWTPs requires sophisticated forecasting and control architectures. Recent research demonstrates a clear evolution toward hybrid AI-based forecasting models that combine deep learning with optimization algorithms ^{IturraldeCarrera2025</sup> . These advanced models significantly outperform classical deterministic methods in handling the highly dynamic and non-linear conditions of real-world PV generation.</p>} Machine learning ensemble algorithms such as XGBoost, LightGBM, and CatBoost have emerged as foundational tools for high-accuracy solar power prediction ^{Nguyen2025</sup> . A critical finding is that humidity and ambient temperature emerge as the most influential factors affecting PV module efficiency, particularly in tropical and humid climates.</p>} Integrated Energy Management</h2> Effective PV system integration requires a multi-layered control hierarchy. The foundational layer is Maximum Power Point Tracking (MPPT), which continuously adjusts the DC-DC converter to extract maximum available power from the solar array. A critical technical constraint in many jurisdictions is the Zero Export requirement, which prohibits injecting electrical power back into the grid. To comply, Zero Export Controllers (ZEC) operate on near-instantaneous feedback loops ^{Alnawafah2025</sup> .</p>} Battery Energy Storage Systems provide flexibility to address both PV intermittency and zero-export constraints. Advanced control algorithms optimize charging during low-cost periods and discharging during peak tariff hours ^{Hvala2025</sup> .</p>} Conclusion</h1> The successful integration of solar PV systems into Malaysian WWTPs needs a strategic of advanced forecasting. By combining accurate generation forecasting with sophisticated demand-side prediction, facilities can achieve near energy-autonomous operation while positioning themselves as flexible power assets responding to grid demands.</p> In the future research we need to explore the details implementation of predictive modelling and control algorithms in WWTPs, as well as conducting in-depth literature reviews on case studies of existing energy predictive systems in WWTP.</p> Gradient Descent Algorithm Explained 2025-11-30T00:00:00+00:00 Gradient descent is simple to state and easy to get wrong in practice, almost always through the learning rate. This note works through the rule itself and then through what happens when the learning rate is too small, reasonable, and too large, with small examples you can follow by hand.</p> The Update Rule</h2> Consider a differentiable function $f: \mathbb{R}^n \to \mathbb{R}$. The gradient of $f$ at a point $x \in \mathbb{R}^n$ is denoted as $\nabla f(x)$, which is a vector of partial derivatives. The gradient points in the direction of steepest ascent. To find a local minimum, gradient descent simply moves the other way:</p> $$x_{k+1} = x_k - \alpha \nabla f(x_k),$$</p> where:</p> $x_k$ is the current point,</li> $\alpha$ is the learning rate, where $\alpha > 0$,</li> $\nabla f(x_k)$ is the gradient of $f$ at point $x_k$.</li> </ul> We simulate two scenarios, one where the loss has a positive gradient and one where it is negative.</p> Scenario 1: Positive Gradient</h3> Let's consider a simple quadratic function:</p> $$f(x) = x^2 + 4x + 4.$$</p> The gradient of this function is:</p> $$\nabla f(x) = 2x + 4.$$</p> Starting from an initial point, say $x_0 = 0$, and choosing a learning rate $\alpha = 0.1$, we can apply the gradient descent update rule iteratively:</p> Compute the gradient at the current point: $\nabla f(x_0) = 2(0) + 4 = 4$.</li> Update the point: $x_1 = x_0 - 0.1 \cdot 4 = 0 - 0.4 = -0.4$.</li> Repeat the process for a number of iterations.</li> </ol> For the first 5 iteration, we can tabulate the results as follows:</p> Iteration ($k$)</th> Current Point ($x_k$)</th> Gradient ($\nabla f(x_k)$)</th> Updated Point ($x_{k+1}$)</th></tr></thead> 0</td> 0.0</td> 4.0</td> -0.4</td></tr> 1</td> -0.4</td> 3.2</td> -0.72</td></tr> 2</td> -0.72</td> 2.56</td> -0.976</td></tr> 3</td> -0.976</td> 2.048</td> -1.1808</td></tr> 4</td> -1.1808</td> 1.6384</td> -1.34464</td></tr> </tbody></table> We can observe that the gradient os loss function from positive value is moving backward each time step, and the first step size is larger but it gradually decreases as we approach the minimum point. This is the brilliant part of gradient descent, as it automatically take larger steps when we are far from the minimum and smaller steps as we get closer to the minimum.</p> We can visualize the process using a simple plot:</p> import</span> numpy</span> as</span> np</span></span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span> </span> #</span> Define the function and its gradient</span></span> def</span> f</span>(</span>x</span>)</span>:</span></span> return</span> x</span></span>2</span> +</span> 4</span></span>x</span> +</span> 4</span></span> def</span> grad_f</span>(</span>x</span>)</span>:</span></span> return</span> 2</span></span>x</span> +</span> 4</span></span> </span> #</span> Gradient Descent parameters</span></span> alpha</span> =</span> 0.1</span></span> x0</span> =</span> 0</span></span> iterations</span> =</span> 20</span></span> </span> #</span> Store the points</span></span> x_points</span> =</span> [</span>x0</span>]</span></span> for</span> _</span> in</span> range</span>(</span>iterations</span>)</span>:</span></span> grad</span> =</span> grad_f</span>(</span>x_points</span>[</span>-</span>1</span>]</span>)</span></span> x_new</span> =</span> x_points</span>[</span>-</span>1</span>]</span> -</span> alpha</span> </span> grad</span></span> x_points</span>.</span>append</span>(</span>x_new</span>)</span></span> </span> #</span> Plotting</span></span> x</span> =</span> np</span>.</span>linspace</span>(</span>-</span>5</span>,</span> 1</span>,</span> 100</span>)</span></span> y</span> =</span> f</span>(</span>x</span>)</span></span> plt</span>.</span>plot</span>(</span>x</span>,</span> y</span>,</span> label</span>=</span>'</span>f(x) = x^2 + 4x + 4</span>'</span>)</span></span> plt</span>.</span>scatter</span>(</span>x_points</span>,</span> f</span>(</span>np</span>.</span>array</span>(</span>x_points</span>)</span>)</span>,</span> color</span>=</span>'</span>red</span>'</span>)</span></span> plt</span>.</span>plot</span>(</span>x_points</span>,</span> f</span>(</span>np</span>.</span>array</span>(</span>x_points</span>)</span>)</span>,</span> color</span>=</span>'</span>red</span>'</span>,</span> linestyle</span>=</span>'</span>--</span>'</span>,</span> label</span>=</span>'</span>Gradient Descent Path</span>'</span>)</span></span> plt</span>.</span>title</span>(</span>'</span>Gradient Descent on f(x)</span>'</span>)</span></span> plt</span>.</span>xlabel</span>(</span>'</span>x</span>'</span>)</span></span> plt</span>.</span>ylabel</span>(</span>'</span>f(x)</span>'</span>)</span></span> plt</span>.</span>legend</span>(</span>)</span></span> plt</span>.</span>grid</span>(</span>)</span></span> plt</span>.</span>show</span>(</span>)</span></span></code></pre> After 20 iterations, the points converge towards the minimum point at $x = -2$. We can see how the points move along the curve of the function, gradually approaching the minimum.</p> </p> Scenario 2: Negative Gradient</h3> Now, let's consider a function with a negative gradient: $$f(x) = -x^3 + 4x^2 - 4.$$</p> The gradient of this function is: $$\nabla f(x) = -3x^2 + 8x.$$</p> Similar to what we have done on the previous example, we start from an initial point, say $x_0 = 0$, and choosing a learning rate $\alpha = 0.01$, we can apply the gradient descent update rule iteratively:</p> Compute the gradient at the current point: $\nabla f(x_0) = -3(0)^2 + 8(0) = 0$.</li> Update the point: $x_1 = x_0 - 0.01 \cdot 0 = 0$.</li> Repeat the process for a number of iterations.</li> </ol> For the first 5 iterations, we can tabulate the results as follows:</p> Iteration ($k$)</th> Current Point ($x_k$)</th> Gradient ($\nabla f(x_k)$)</th> Updated Point ($x_{k+1}$)</th></tr></thead> 0</td> 1.0000</td> 5.0000</td> 0.9500</td></tr> 1</td> 0.9500</td> 4.7175</td> 0.9028</td></tr> 2</td> 0.9028</td> 4.4533</td> 0.8583</td></tr> 3</td> 0.8583</td> 4.2057</td> 0.8162</td></tr> 4</td> 0.8162</td> 3.9734</td> 0.7765</td></tr> </tbody></table> In this scenario, we can observe that the gradient of loss function from negative value is moving forward each time step, and the first step size is larger but it gradually decreases as we approach the minimum point. Similar to the previous scenario, gradient descent automatically adjusts the step size based on the distance from the minimum. Similarly, we can visualize the process using a simple plot:</p> import</span> numpy</span> as</span> np</span></span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span> </span> #</span> Define the function and its gradient</span></span> def</span> f</span>(</span>x</span>)</span>:</span></span> return</span> -</span>x</span>*</span>3</span> +</span> 4</span></span>x</span>*</span>2</span> -</span> 4</span></span> def</span> grad_f</span>(</span>x</span>)</span>:</span></span> return</span> -</span>3</span></span>x</span>*</span>2</span> +</span> 8</span></span>x</span></span> </span> #</span> Gradient Descent parameters</span></span> alpha</span> =</span> 0.01</span></span> x0</span> =</span> 1</span></span> iterations</span> =</span> 40</span></span> </span> #</span> Store the points</span></span> x_points</span> =</span> [</span>x0</span>]</span></span> for</span> _</span> in</span> range</span>(</span>iterations</span>)</span>:</span></span> grad</span> =</span> grad_f</span>(</span>x_points</span>[</span>-</span>1</span>]</span>)</span></span> x_new</span> =</span> x_points</span>[</span>-</span>1</span>]</span> -</span> alpha</span> </span> grad</span></span> x_points</span>.</span>append</span>(</span>x_new</span>)</span></span> </span> #</span> Plotting</span></span> x</span> =</span> np</span>.</span>linspace</span>(</span>-</span>1</span>,</span> 3</span>,</span> 100</span>)</span></span> y</span> =</span> f</span>(</span>x</span>)</span></span> plt</span>.</span>plot</span>(</span>x</span>,</span> y</span>,</span> label</span>=</span>'</span>f(x) = -x^3 + 4x^2 - 4</span>'</span>)</span></span> plt</span>.</span>scatter</span>(</span>x_points</span>,</span> f</span>(</span>np</span>.</span>array</span>(</span>x_points</span>)</span>)</span>,</span> color</span>=</span>'</span>red</span>'</span>)</span></span> plt</span>.</span>plot</span>(</span>x_points</span>,</span> f</span>(</span>np</span>.</span>array</span>(</span>x_points</span>)</span>)</span>,</span> color</span>=</span>'</span>red</span>'</span>,</span> linestyle</span>=</span>'</span>--</span>'</span>,</span> label</span>=</span>'</span>Gradient Descent Path</span>'</span>)</span></span> plt</span>.</span>title</span>(</span>'</span>Gradient Descent on f(x)</span>'</span>)</span></span> plt</span>.</span>xlabel</span>(</span>'</span>x</span>'</span>)</span></span> plt</span>.</span>ylabel</span>(</span>'</span>f(x)</span>'</span>)</span></span> plt</span>.</span>legend</span>(</span>)</span></span> plt</span>.</span>grid</span>(</span>)</span></span> plt</span>.</span>show</span>(</span>)</span></span></code></pre> After 20 iterations, the points converge towards the minimum point at approximately $x = 0$. We can see how the points move along the curve of the function, gradually approaching the minimum. We plotted the gradient descent path on the function curve:</p> </p> Proof of Convergence</h2> To prove the convergence of the gradient descent algorithm, we need to show that the sequence of points generated by the algorithm converges to a local minimum of the function $f(x)$. We assume that $f$ is a convex function with Lipschitz continuous gradients, meaning there exists a constant $L > 0$ such that for all $x, y \in \mathbb{R}^n$, $$\|\nabla f(x) - \nabla f(y)\| \leq L \|x - y\|.$$</p> Under convexity alone, we show that the rate of convergence is sublinear. Specifically, we can show that after $k$ iterations, the function value satisfies:</p> $$ f(x_k) - f(x^) \leq \frac{L \|x_0-x^\|^2 }{2k}, $$</p> where $x^$ is the global minimum point of $f$. This indicates that as the number of iterations $k$ increases, the function value approaches the minimum value at a rate inversely proportional to $k$.</p> This completes the proof of convergence for the gradient descent algorithm under the assumptions of convexity and Lipschitz continuous gradients. The algorithm effectively finds a local minimum of the function $f(x)$ by iteratively updating the points in the direction of the steepest descent.</p> But all of this hangs on the choice of learning rate $\alpha$. Too large and the algorithm overshoots the minimum and diverges. Too small and convergence crawls. There are various ways to adapt the learning rate during optimization, like schedules and optimizers such as Adam and RMSprop, which I may cover in a future post.</p> Known Issues of Gradient Descent</h2> While gradient descent is a powerful optimization algorithm, it does have some known issues. In multivariate functions, the presence of saddle points can affect the convergence. Saddle points are points where the gradient is zero, but they are neither local minima nor local maxima. In high-dimensional spaces, saddle points are more prevalent than local minima, and gradient descent can get stuck at these points, leading to slow convergence or failure to find the global minimum. A popular example is the function $f(x, y) = x^2 - y^2$, which has a saddle point at $(0, 0)$. The direction vector at this point is zero, and gradient descent may struggle to escape this point.</p> Gradient Descent Saddle Point</figcaption> </figure> To mitigate the issues with saddle points, various techniques can be employed, such as adding noise to the gradients, using momentum-based methods, or employing second-order optimization methods that consider the curvature of the function, which we can explore in future discussions. But overall, gradient descent remains a fundamental and widely used optimization algorithm in machine learning and various other fields.</p> Conclusion</h2> That is gradient descent: a one-line update rule that ends up as the backbone of most of machine learning.</p> It's remarkable how such a simple iterative process can lead optimize almost any complex function in real life applications. I truly cannot appreciate enough the beauty of this elegant mathematical concept.</p> For those who are interested to learn more about gradient descent, I highly recommend watching the following video by StatQuest, which provides an excellent visual explanation of the algorithm:</p> </iframe> </div> Gradient Descent, Step-by-Step \| StatQuest</figcaption> </figure> I hope this post has provided a clear understanding of the gradient descent.</p> My Gallery of Talentbank Boardroom Challenge 2025 2025-11-14T00:00:00+00:00 This post is a compact gallery plus debrief from the Talentbank Boardroom Challenge 2025. I keep the narrative focused on preparation, presentation decisions, and the specific lessons carried into later projects.</p> </figure> </figure> </figure> </figure> </figure> </figure> Reinforcement learning practices in healthcare applications 2025-10-21T00:00:00+00:00 This note reviews practical reinforcement learning use cases in healthcare and the constraints that matter in deployment. The challenge is that policy learning in clinical settings is high-stakes, partially observed, and often offline. I summarize where RL is promising and where reliability and safety dominate design choices.</p> In healthcare applications, artificial intelligence (AI) plays a crucial role in transforming patient care, diagnostics, and treatment planning to make healthcare more efficient and effective. However, if AI is used improperly, it may leads to worse outcomes rather than improved ones.</p> In the subset of AI, reinforcement learning (RL) has shown great promise in optimising sequential decision-making processes, which are common settings in healthcare industry. However, applying RL in healthcare settings requires careful consideration of several important practices to ensure a safety outcome. To illustrate the pitfalls of reinforcement learning, we consider the sepsis management, which remains wide uncertainty in the way clinicians make decisions.</p> In the context of sepsis, a history may include a patient's vital signs, laboratory results, administered treatments, and other relevant clinical information over time. The actions could involve decisions such as administering fluids, vasopressors, or antibiotics at different time points. The rewards are typically defined based on patient outcomes, such as survival rates, length of hospital stay, or improvement in clinical scores. Note that defining ideal sepsis resuscitation strategies is challenging due to the complex and dynamic nature of the condition, as well as the variability in patient responses to treatments, therefore it is not straightforward to define short-term rewards for each action taken.</p> Here are three fundamental concerns when applying reinforcement learning in healthcare:</p> Is the AI given access to all variables that influence decision making?</h1> RL agent can only ook at the recorded data, however there are much more information and context that should be taken into consideration. Failing to consider all variables may result in esitmates that are confounded by spurious correlation.</p> For instance, severely sick septic patients may receive fluids earlier than healthier patients yet have worse outcomes, which is clearly because of them being sicker in the first place, not because of fluids worsen the outcomes. Therefore, it is important to considers of pissble confounding factors, which even more than what is required for standard prediction studies, as the sequential nature of the problem could possibly lead to confounding effects in both long term and short term.</p> How big was that big data?</h1> This is relatively straightfoward. For any AI training, its necessary to feed the model an adequate amount of useful information, so is RL model. Logically, for RL model to evaluate a new policy, it needs to find a long, continuous sequence of decisions in the historical data that matches its new policy.</p> In clinical trials, the mismatches between new treatment policy against historical data, also known as off-policy evaluation, the effective sample size can become small. The mismatches grow with the number of decisions in a patient's history. For one sepsis study, a cohort of 3,855 patients yielded an effective sample size of only a few dozen. Therefore, instead of exploring new treatment approaches, observational data shall be used only to refining existing practices.</p> Will the AI behave prospectively as intended?</h1> One of the core elements in the feedback loop of every RL is rewards. However, if the design of reward function is not handled properly (e.g. error in formulation, data processing etc.), the momdel will eventually laed to poor decisions.</p> Often, an overly simple reward function may neglect the long-term effects. For instance, rewarding only blood pressure targets may result in agent that harms long-term benefit by dosing excessive vasopressors to patients. Additionally, the learned policy might decay after a period of time if there's changes in the treatment standards.</p> Therefore, it is neccessary to use interpretable machine learning to interrogate and assess whether learned policies will behave as intended in a prospective clinincal setting</p> Conclusion</h1> In the end, although RL offers such promising opportunities to optimising sequential treatments in medical industry, we shall be cautious in deploying into production and requires due diligence to safely realise its potential in revolutionalising this life-saving industry.</p> Guidelines for Reinforcement Learning in Healthcare</figcaption> </figure> In the end, I would like to express my gratitude to Omer Gottesman et al. for providing such practical viewpoint in standardising the application of reinforcement learning in clinical settings.</p> References:</p> Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., & Celi, L. A. (2019). Guidelines for reinforcement learning in healthcare. Nature Medicine, 25(1), 16–18. https://doi.org/10.1038/s41591-018-0310-5</a></li> </ul> Action-value methods with incremental step size in reinforcement learning 2025-10-17T00:00:00+00:00 This note derives the incremental update rule for action-value estimation in k-armed bandits and explains why it is preferable to recomputing full averages. The problem is memory and compute cost when rewards accumulate over time. By the end, you get a practical update equation you can implement directly in RL experiments.</p> Consider any k-armed bandit problem</a>, we consider each actions taken as pulling an arm of a slot machine, and the machine gives us a reward based on the action taken. We denote the action selected at time step $t$ as $A_t$, and the correspondiong reward received as $R_t$. Among the $k$ actions, the expected value of each action $a$ is denoted as $q_(a)=\mathbb{E}[R_t\|A_t=a]$, which is also known as the value</em> of that action $a$. It is rationale to choose the action with the highest value, however in the practical applications, the value of each action is unknown. Therefore, we need to estimate the value of each action, denoted as $Q_t(a)$, read as the estimated value of action $a$ at time step $t$. The fundamental goal is to find such $Q_t(a)$ that is as close as possible to the $q_(a)$. It is commonly known as the true value</em> by community.</p> To estimate the values of each action, which we collectively call as the action-value methods</em>, we can use the sample-average method to update the estimated value of action $a$ at time step $t$ as follows:</p> $$Q_{t}(a)={{\text{sum of rewards when $a$ taken prior to $t$}} \over {\text{number of times $a$ taken prior to $t$}}}.$$</p> This method simply takes the average of all the rewards received when action $a$ is taken prior to time step $t$. To simplify the notation, we focus on a single action. We denoted $R_i$ as the reward received the $i$-th time of this action is taken, and $n$ as the number of times action $a$ is taken prior to time step $t$. Logically, we can let $Q_n$ denote the estimate of its action value after it has been taken $n-1$ times. Therefore, we can rewrite the update rule as follows:</p> $$Q_n = \frac{\sum_{i=1}^{n-1}R_i}{n-1} = \frac{R_1+R_2+\ldots+R_{n-1}}{n-1}.$$</p> As the number of times of the action, $n$ increases, the obvious wat to update the estimate is to recalculate the average by summing up all the previous rewards and dividing it by $n-1$. However, as the number of times of the action increases to a large number, this method becomes progressively more expensive as we need to store all the previos rewards and recalculate the sum every time we need to update the estimate.</p> But is there a better way to update the estimate without storing all the previous rewards? The answer is yes. We devise the incremental formulas for updating the estimate by</p> $$ \begin{align} Q_{n+1} & = \frac{1}{n}\sum_{i=1}^{n}R_i \\ & = \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1}R_i\right) \\ & = \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1}R_i\right) \\ & = \frac{1}{n}\left(R_n + (n-1)Q_n\right) \\ & = \frac{1}{n}\left(R_n+nQ_n-Q_n \right) \\ & = Q_n + \frac{1}{n}[R_n - Q_n]. \end{align} $$</p> This incremental formula allows us to update the estimate $Q_n$ to $Q_{n+1}$ by only using the most recent reward $R_n$ and the previous estimate $Q_n$, without the need to store all the previous rewards. The term $\frac{1}{n}$ serves as the step size, which decreases as $n$ increases, ensuring that the estimate converges to the true value over time.</p> Even in $n=1$, we can still obtain $Q_2 = R_1$ for arbitrary initial estimate $Q_1$. In this case, the initial estimate $Q_1$ is completely ignored after the first update, as it should be. In processing the $n$th reward, the estimate is adjusted by a fraction of the error term $[R_n - Q_n]$, which is the difference between the received reward and the current estimate. This adjustment is scaled by the step size $\frac{1}{n}$, which ensures that as more data is collected, the updates become smaller, allowing the estimate to stabilize around the true value. Note that the step size here is not constant, it decreases as the number of times of the action increases.</p> Back to the bandit problem, the proposed simulation in for pseudo-code is as follows:</p> Bandit Problem with Incremental Step Size</summary> Initialize</span> Q</span>(</span>a</span>)</span> arbitrarily</span> for</span> all</span> actions</span> a</span></span> For</span> each</span> time</span> step</span> t</span> = </span>1</span>,</span> 2</span>,</span> ...</span></span> Select</span> action</span> A_t</span> using</span> a</span> policy</span> derived</span> from</span> Q</span> (</span>e</span>.</span>g</span>.</span>,</span> ε</span>-</span>greedy</span>)</span></span> Take</span> action</span> A_t</span> and</span> observe</span> reward</span> R_t</span></span> Update</span> the</span> estimate</span> Q</span>(</span>A_t</span>)</span> using</span>:</span></span> n</span> = </span>number</span> of</span> times</span> action</span> A_t</span> has</span> been</span> taken</span> prior</span> to</span> time</span> t</span></span> Q</span>(</span>A_t</span>)</span> = </span>Q</span>(</span>A_t</span>)</span> +</span> (</span>1</span>/</span>n</span>)</span> </span> [</span>R_t</span> -</span> Q</span>(</span>A_t</span>)</span>]</span></span> End</span> For</span></span></code></pre> </div> </details> In the end of this post, we derived the incremental step size method for updating the aciton-value estimates which extensively applied in reinforcement learning. This method is computationally efficient as it does not require storing all previous rewards, and it ensures convergence to the true action values over time.</p> Reinforcement Learning: An Introduction</figcaption> </figure> I want to express my gratitude to Sutton and Barto for their excellent book Reinforcement Learning: An Introduction</a> that provides a comprehensive introduction to the concepts and algorithms of reinforcement learning.</p> References:</p> Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf</a></li> </ul> A Small Talk About Hackathons 2025-08-07T00:00:00+00:00 This post is a short reflection on hackathon culture from a participant perspective: what helps teams learn fast, where teams usually waste time, and how to keep projects grounded under deadline pressure.</p> Along the way, I've met people who are far more experienced, far more knowledgeable than me. And honestly, I can't compete with them. It's easy to feel small in those moments.</p> For anyone who's been through hackathons, you'll know. The amount of time, energy, effort you need to commit is just INSANE. Most competitions require you to go through multiple stages: prelim round, and sometimes a semi-final, and then the final round. Every round is like a mini-marathon, endless brainstorming, last-minute changes, and almost no sleep just to push through and deliver something that works.</p> In the first few hackathons, I was genuinely excited. It was fun. Every time one ended, I was already looking forward to the next. But after round and round of hackathons, I started to feel the burn. Exhausted, Emptiness. YES, I still learned something new in every game, but the energy, the excitement just started to wear off.</p> Eventually, it all stated to feel a bit empty.</p> There was also this lingering thought back in my mind: "Why am I comparing myself to people who are fully in this industry, who live and breathe in tech every single day, when I'm still just a student trying to explore things outside my field?"</p> There's still one more hackathon coming up, and I'll give it what I can. but after that, I really need a break. A proper one. Not just from hackathons, but from the constant cycle of proving myself. I need to breathe, reset and work on my inner health, mentally and emotionally.</p> Not saying I'm quitting hackathons forever. Not at all. But I know I need some time to focus more on myself, rebuild myself, and come back stronger.</p> Thanks for reading until here. Sometimes, the best thing you can do for your growth is step back, realign, and then go again.</p> Dead Internet Theory #1 2025-07-22T00:00:00+00:00 This post is a personal reflection on authenticity online: what feels different now, why AI-generated social content often feels hollow, and where I might be overreacting. The goal is not to claim a grand theory, but to document a concrete shift in reading experience across LinkedIn and Reddit.</p> </p> I do admit I used ChatGPT for my content in the past, but I realised that the content is not really what I’ve done before, the experience is basically “artificial”. I don’t deny AI as a productivity tool, but sometimes you just can’t tell the existence of the content written there. There’s a coldness, an emptiness that creeps in when scrolling through these posts, making it feel like shouting into a void where no real person listens.</p> It is really different nowadays to scroll through social media, LinkedIn, and Reddit. You don’t feel people there. It makes me really feel like the dead Internet Theory is here, and everything is full of bot activity and automatically generated content manipulated by algorithms. The authenticity of online interactions seems to be fading, they replaced by automated perfection that feels disturbingly hollow.</p> Chatbots or AI are really good for proofreading, but they are still only good at proofreading. Soon I think they will no longer have real content or real ideas when people are writing their experiences, their ideas; they just become full of BS nowadays. Now I even wish to find some long-ass article that is awfully organised, I find some fun to read through even though it was not good, but I can find authenticity there.</p> Has anyone noticed this? You could share them with me or offer me a perspective I haven’t considered before. I’d be more than happy to discuss it. I think we’ll might have another follow-up episode on this.</p> Share me your thoughts: contact@jienweng.com</a></p> Making Deepseek R1 ChatBot 2024-12-30T00:00:00+00:00 This post documents a small DeepSeek-R1 chatbot build: why I chose the model, what setup decisions mattered, and what worked in practice. Instead of focusing on AI industry drama, I keep the write-up centered on implementation choices and takeaways for future iterations.</p> </a></p> But forget the drama for a second, because the best part? Deepseek is open-source. That’s a huge win for the AI community. No more being locked behind API paywalls or waiting for some corporate overlord to decide what we can or can’t do. It’s out there, free to tinker with, and you bet I had to try it out for myself.</p> So, I went ahead and do something I wanted to do for soooo long -- built a chatbot. It’s not packed with fancy features (yet), but through this little experiment, I’ve discovered some pretty interesting things about how the Deepseek R1 model works. You can try it out live here</a>.</p> btw, we won’t dive into the technical aspects just yet—that’s coming up in the next section! Stay tuned for more details on how these improvements will work behind the scenes.</p> The Unique "Thinking" Approach</h3> What blows my mind the most about this whole setup is how I managed to separate the model’s thinking process from its final response. Most chatbots out there? They just spit out an answer, and you have no idea what’s happening behind the scenes. But with this, you can actually see how the model thinks through a problem before giving an answer. It’s like watching an AI have an inner monologue, refining its thoughts before speaking. And honestly? I’ve never seen this before in any LLMs I’ve used.</p> At first, I didn’t even plan for this feature—it just happened while I was testing out different ways to improve response quality. I noticed that the model was generating some hidden reasoning steps before its final output. Instead of discarding them, I figured, Why not show them? And once I did, it was a game-changer. It made the AI feel so much more transparent—almost like it was thinking out loud.</p> </a></p> For example, if you ask it something like, “What do you think about climate change in Malaysia?”, you won’t just get a final answer out of nowhere. You’ll actually see the model go through a step-by-step breakdown of its thought process:</p> Breaking down the question components</li> Evaluating current knowledge</li> Forming logical connections</li> Synthesizing a comprehensive response</li> </ol> After seeing the model’s thinking process, what really stands out to me is how structured its response is. It doesn’t just throw out some generic take on climate change—it actually analyzes the question, breaks it down into different angles, and then builds a well-organized answer.</p> </a></p> That said, while the response does sound solid, there are some oddities that make me wonder what’s going on under the hood. For example, it mentions “the subtropical Andaman and Nicobar Islands”—which, uh, aren’t even part of Malaysia. Also, “Ch bamboo” initiative? Never heard of that one. These small but noticeable mistakes show that while the model is good at structuring its answers, it still struggles with factual accuracy.</p> But that’s exactly what makes having a visible thought process so useful. Instead of just blindly trusting AI responses, we can now see how the model arrives at its conclusions—which means we can spot errors more easily. If it had hallucinated</strong> this stuff in a normal chatbot, I might not have even noticed. But because I can watch it reason through the problem, I can tell where things might be going wrong.</p> </a></p> This kind of transparency is what makes AI feel less like a magic black box and more like an actual tool that we can guide, correct, and refine. And that’s honestly what excites me the most about this project.</p> Deployment Specifications</h3> The chatbot is currently hosted on Hugging Face Spaces, running on a basic-tier instance, which means it’s not exactly a powerhouse but still gets the job done. Here’s what it’s running on:</p> CPU: 2 vCPUs</li> RAM: 16GB</li> Storage: Basic instance storage</li> Framework: Gradio</li> Inference Optimization: FP16 quantization</li> Average Response Time: 2-3 seconds</li> Concurrent Users Supported: Up to 10</li> </ul> You might notice that the live preview</a> here can be a bit slow while generating responses. That’s because the hardware isn’t optimized for LLM inference, so it’s working with some limitations. Hope you can bear with it! 😆</p> If you enjoy the project and want to see it run smoother, you can consider sponsoring me</a>. Who knows? With enough support, I might upgrade the resources for future projects and push this even further :D</p> Efficient Model Architecture</h3> The chatbot uses the Deepseek R1 Distilled 1.5B model, which is a significantly compressed version of the original 685B parameter model. Despite having only 1.5 billion parameters, it maintains impressive performance for many tasks.</p> Key points about the model:</p> Original model: DeepSeek R1 (685B)</a></li> Distilled version: DeepSeek R1 Distill Qwen 1.5B</a></li> 440x parameter reduction while maintaining core capabilities</li> </ul> </a></p> Impressive Benchmark Results</h3> What’s most fascinating about this model is how well it holds up when compared to much larger models. Despite having far fewer parameters, it manages to outperform some big names in the AI world for certain tasks.</p> </a></p> Outstanding Performance in Key Areas</h4> AIME 2024 (Math Competition)</strong> DeepSeek R1 Distilled: 28.9% Pass@1</li> GPT-4o: 9.3% Pass@1</li> Claude 3.5: 16.0% Pass@1</li> </ul> </li> MATH-500 (Mathematical Reasoning)</strong> DeepSeek R1 Distilled: 83.9% Pass@1</li> GPT-4o: 74.6% Pass@1</li> Claude 3.5: 78.3% Pass@1</li> </ul> </li> Codeforces (Competitive Programming)</strong> DeepSeek R1 Distilled: 954 Rating</li> GPT-4o: 759 Rating</li> Claude 3.5: 717 Rating</li> </ul> </li> </ol> Model Strengths & Limitations</h3> Strengths:</strong></p> Superior reasoning capabilities, especially in mathematics</li> Highly efficient with only 1.5B parameters</li> Effective knowledge distillation from larger models</li> Excellent performance in zero-shot scenarios</li> </ul> Limitations:</strong></p> Lower performance in general coding tasks</li> Potential language mixing issues</li> Sensitivity to prompt formatting</li> Limited performance in broader general knowledge tasks</li> </ul> This balanced perspective shows why I chose this model for my chatbot implementation - it provides exceptional reasoning capabilities while remaining lightweight enough for practical deployment.</p> Try It Yourself</h3> Due to iframe restrictions, you can access the live demo through these methods:</p> Direct Link to Demo</a></li> API Documentation</a></li> Source Code</a></li> </ol> </a></p> Summary</h3> Deepseek has definitely shaken things up in the AI world, and the drama surrounding it is just the tip of the iceberg. Forget the finger-pointing—this move is a win for the AI community, especially since Deepseek is open-source. No more waiting around for companies to decide how we can use AI; now it’s out there for everyone to play with and improve.</p> And as for my little experiment—building a chatbot with the Deepseek R1 model—it’s not feature-packed yet, but it’s definitely been a fun ride. You can try it out live here</a> and see how it works for yourself!</p> Additional Resources</h3> Model Card</a></li> Deployment Guide</a></li> Performance Benchmarks</a></li> Community Discussion</a></li> </ul> Feel free to experiment with the live demo and share your thoughts!</p> References</h3> DeepSeek R1 (685B)</strong> The original DeepSeek R1 model, a large-scale AI model with 685 billion parameters, was the precursor to the distilled 1.5B version used in the chatbot.</em> Source</a></li> DeepSeek R1 Distill Qwen 1.5B</strong> This is the distilled version of the DeepSeek R1 model, compressed to 1.5 billion parameters while retaining core capabilities.</em> Source</a></li> Open R1 Model Architecture</strong> Explore the detailed architecture of the DeepSeek R1 model, showcasing its design and structure.</em> Source</a></li> Medium - Deepseek R1 Distill Qwen 1.5B Performance</strong> A comparison of the performance between Deepseek R1 Distilled and other models, showing its impressive results in multiple domains.</em> Source</a></li> Hugging Face Space - Chatbot Demo</strong> Live demo of the Deepseek R1 chatbot that showcases the model’s response and reasoning capabilities.</em> Source</a></li> Hugging Face - API Documentation</strong> Official API documentation for Hugging Face Spaces, helping developers interact with models and integrate them into applications.</em> Source</a></li> Hugging Face - Source Code</strong> Direct access to the source code of the Deepseek R1 chatbot project on Hugging Face Spaces for those interested in contributing or learning.</em> Source</a></li> Hugging Face - Model Card</strong> Official card for the Deepseek R1 Distilled model, providing details on its functionality and training specifications.</em> Source</a></li> Hugging Face - Deployment Guide</strong> Guidelines for deploying models and applications using Hugging Face Spaces.</em> Source</a></li> Hugging Face - Performance Benchmarks</strong> An overview of the model performance across various tasks and benchmarks, showcasing the strengths and weaknesses of different models.</em> Source</a></li> Hugging Face - Community Discussion</strong> Join the community discussions on Hugging Face, where users can ask questions, share insights, and discuss AI-related topics.</em> Source</a></li> </ol> Eco Finance: A Sustainable Future Prototype 2024-12-29T00:00:00+00:00 This post focuses on the Eco Finance prototype itself: the problem we targeted, the product concept, and what we learned from turning an idea into a demo under hackathon constraints. It is written as a project debrief rather than an event recap.</p> We participated in PayHack 2024, quite a big hackathon event with many talented individuals.</p> Our project, Eco Finance, focuses on the implementation of a carbon tax expected to be released in 2026. Although it initially targets specific industries, it is crucial and trending to implement this for everyone in Malaysia. Read more about the 2026 carbon tax here.</p> European countries are already implementing ESG-centric policies in various industries, such as the automobile industry. These are just preliminary steps; now we want to delve deeper into the subject.</p> </a></p> Understanding the idea</h2> Do you know how much carbon footprint you generate from ordering a Shopee parcel? Or how much carbon you generate by driving to work instead of taking public transport? It's challenging for people to visualize their footprint. Hence, we're here to make this visible to people, increasing their awareness about this issue. According to Visa, 80% of Malaysians are aware of the environmental impact of consumption. With the release of Malaysia's largest payment gateway provider and the latest project linking banks.</p> How it works</h2> The OpenFinance API can seamlessly integrate people's transaction details, allowing us to aggregate a person's transactions and their carbon footprints. The calculation would be the emission factor times the amount, giving us the carbon footprint from those transactions.</p> For example, if the emission factor for a specific merchant category is 0.5 kg CO2 per RM, and a person spends RM 100, the carbon footprint would be 0.5 kg CO2/RM 100 RM = 50 kg CO2.</p> There's an established merchant category code (MCC) in transaction details, where we only need to fine-tune and investigate the actual emission factors for each merchant code. This could be done by collaborating with the Department of Statistics Malaysia (DOSM) to conduct surveys and research among Malaysians. In this project, we are using dummy variables based on this ideology only.</p> Monetizing the Ecosystem</h2> We circulate this whole ecosystem by monetizing it. We extract carbon credits from them using the formula from WORLDMETER, where the baseline of average carbon emission per person in Malaysia is 8 tons per year. With that, we can pool up carbon credits, extracted from the surplus from the calculation. There's a proven market potential with Bursa Carbon Exchange (BCX), established on 9 Dec 2022, which is available to trade carbon credits in Malaysia. Then we sell the carbon credits to major companies in Malaysia like Petronas and Maybank to help them offset their carbon credits. Read more about BCX here</a>.</p> From Petronas</a>, it's evident that their future plan aims for net-zero carbon emission by 2050. This proves there is a market in Malaysia, and more people will enter the market and participate.</p> Encouraging Eco-Friendly Practices</h2> How do we encourage eco-friendly spending habits in Malaysia? We aim to attract more people, even those who are not initially interested in eco-friendly practices, to join us. We can make Malaysia greener and more sustainable, at least in the sense of ESG. We choose to reward users with healthy spending habits in terms of eco-friendly spending and reward them with cash for being environmentally friendly.</p> </p> Sustainable Business Model</h2> We can summarize the business circulation here and make it sustainable as well, where we can become self-sustaining:</p> Collect carbon credits from users</li> Pool up carbon credits</li> Certify carbon credits with Bursa Carbon Exchange (BCX)</li> Sell carbon credits to companies who need them</li> Reward users to encourage eco-friendly spending habits.</li> </ol> Basically that's it.</p> Project Random Thingy</h2> Now, it's about the random thingys of the project. We have the project hosted at this link. Although it is not fully complete, it serves as a prototype and is yet to be an MVP. Considering we had only 24 hours to complete this from idea to execution, I mean, it's good for a first-timer. Right...?</p> You can access the hosted project prototype here</a>.</p> Feel free to browse through it. If you have any questions, please email me, and I'll personally explain it to you. You can also see the admin page by accessing here</a> or by changing the /dashboard</code> to /admin</code>. It looks something like this:</p> </p> Feel free to browse through it.</p> Wrap-Up</h2> In conclusion, Eco Finance aims to make carbon footprints visible to individuals, encouraging eco-friendly spending habits and contributing to a greener Malaysia. By monetizing carbon credits and rewarding users, we create a sustainable ecosystem that benefits both the environment and the economy. It's really kesian that we couldn't make it to final though T.T</p> Hope you guys like it :D</p> Oh btw! Once again.. You can read the full story here</a>.</p> First Physical Hackathon Experience 2024-12-02T00:00:00+00:00 This post records what our first physical hackathon taught us as a math-heavy team entering a software-first environment. I focus on concrete lessons from ideation, mentoring, and pitching that we can reuse in future competitions.</p> I found myself staring at my phone, thumb hovering over the share button. "Am I really qualified for this?" I thought to myself. "What if we make fools of ourselves?" The doubts crept in like unwanted guests. But then another voice, stronger and more determined, pushed back: "When else will we get a chance like this? We might not be coders, but we know how to solve problems. Isn't that what hackathons are really about?"</p> I reached out to my fellow mathematics coursemates: Janice, Roius, and Gwyn. "Hey, want to do something crazy?" I asked, half expecting them to laugh it off. To my surprise, their responses came quickly, filled with enthusiasm despite (or maybe because of) our collective inexperience. Only 2 of us had ever participated in a hackathon before, and Gwyn had never even written a line of code. But there we were, four mathematics students from UTAR, signing up for one of the most competitive hackathons in the country.</p> The looks we got when we arrived were priceless. "Are you guys from Computer Science?" someone asked, eyeing our team with curiosity. We exchanged glances and grinned. "Nope, we're Math students, haha..." The mixture of surprise and skepticism on their faces was something I'll never forget. In those moments, our outsider status felt both terrifying and weirdly empowering.</p> </p> November 30th marked our first day, and it was a blur of ideation and learning. We came up with an innovative idea: creating an app to visualize transaction carbon footprints, with the ability to pool and trade carbon footprint surpluses with companies in need. On paper, it sounded promising—a perfect blend of fintech and environmental consciousness.</p> </p> Then came the intense mentoring sessions with Johan Nasir. He didn't hold back. "What happens if the carbon credits are manipulated?" he'd challenge. "How do you ensure the authenticity of the footprint data?" Another round of rethinking. Each session felt like an intense rotan session—tough love at its finest. He'd poke holes in our solutions, push us to think deeper, and force us to confront real-world problems that actually needed solving.</p> The week before the hackathon was a rollercoaster. Competing against almost 100 teams from across Malaysia, we were shocked to make it to the Top 32. When we saw Johan's name among our judges and received the news of our advancement, our excitement was through the roof. But reality quickly set in—we had a major problem. None of us had real web development experience. My knowledge was limited to Python, SQL, and vanilla HTML/CSS/JavaScript.</p> With just four days between Tuesday and Friday, we had to make crucial technical decisions while juggling our internships. After intense research, we settled on Vue.js and Flask. Janice even traveled all the way from JB to KL for this. Every evening after our internships, we'd dive into tutorials, trying to absorb as much as we could about our chosen tech stack.</p> </p> The hackathon itself was intense. When exhaustion hit, we took turns napping wherever we could. I couldn't get a tent, so I made do with a random bench for quick 4-hour power naps before jumping back into coding. Everything seemed to be going smoothly until the morning of the submission. At 8:30 AM, just after breakfast and 90 minutes before the deadline, our backend crashed—the information couldn't be parsed properly. In a desperate move, we had to hardcode some components just to make the submission deadline at 10 AM.</p> </p> While we didn't make it to the Top 10, watching the final pitches was an eye-opening experience. The winning teams showcased solutions that were not just technically impressive but also deeply thoughtful about real-world implementation. One team's blockchain-based remittance system particularly stood out - their attention to regulatory compliance and market research was incredible. "We should have done more market validation," I thought to myself. "Next time, we need to focus not just on the technical solution but on the whole business case."</p> The top teams also demonstrated masterful presentation skills. Their pitches weren't just about features - they told compelling stories about why their solutions mattered. Each slide was carefully crafted, each demo was flawlessly executed, and their responses to judges' questions showed deep understanding of both technical and business aspects. I made mental notes: "Practice the pitch more. Know your numbers. Be ready for any question."</p> Despite not making the finals, I did win a Samsung monitor in the lucky draw, a small consolation that brought some laughs to our tired team.</p> </p> Looking back now, the sleepless nights and endless debugging sessions blur together, but certain moments stand crystal clear: the late-night breakthrough when our first feature finally worked, the proud smile on Johan's face during our final presentation, and most importantly, the unshakeable bond formed between four mathematicians who dared to dream.</p> </p> To Janice, Roius, and Gwyn: thank you for taking this leap of faith with me. For believing that our mathematical minds could contribute something meaningful to the tech world. To Johan: your guidance went beyond mentorship—you showed us that innovation comes from daring to be different. And to PayNet and JomHack: thank you for creating a space where even mathematics students could discover their potential in technology.</p> They say the best stories come from stepping out of your comfort zone. Well, we didn't just step—we took a giant leap. And while our first hackathon journey has ended, something tells me this is just the beginning of our adventure in the tech world. The equations and formulas we've studied for years are no longer just abstract concepts—they're tools waiting to be applied in the vast playground of technology.</p> After all, who says mathematicians can't be hackers too? Sometimes the best innovations come from those who dare to cross the boundaries between disciplines, who bring fresh perspectives to old problems. And maybe, just maybe, that's exactly what the tech world needs more of.</p>

Jien Weng

The Fokker-Planck Equation Explained, from an SDE to the Density it Evolves

2026-07-01T00:00:00+00:00

Most explanations of the Fokker-Planck equation start too late. They write down a partial differential equation for a density, call one term "drift" and the other "diffusion", and move on. That is correct, but it hides where the equation comes from.

The cleaner way to think about it is to start one step earlier, with a single stochastic trajectory. A stochastic differential equation describes one random path. But one path tells you almost nothing. What we usually want is the distribution: given where the system started, where is it likely to be at time $t$? The Fokker-Planck equation is the deterministic law that this distribution obeys.

So the real chain is:

$$ \text{SDE} \to \text{ensemble of paths} \to \text{density } p(x,t) \to \text{Fokker-Planck PDE} \to \text{stationary law}. $$

Once that chain is visible, the equation stops looking like a definition to memorize. It becomes the obvious bookkeeping for how probability mass flows: drift transports it, diffusion spreads it.

This note walks the full path. We start from a concrete SDE and simulate it, watch a cloud of paths turn into a density, derive the Fokker-Planck equation from Ito's lemma, and then solve it from scratch in Python. We finish with an Ornstein-Uhlenbeck example whose answer we know in closed form, so we can check three independent routes against each other.

Run it yourself. Every code block below is collected in a runnable Jupyter notebook: download fokker-planck-sde.ipynb</code></a>. It only needs numpy</code> and matplotlib</code> (pip install numpy matplotlib</code>), and runs top to bottom. </blockquote>
Start with a Single Trajectory</h2> A stochastic differential equation in one dimension has the form $$ dX_t = \mu(X_t, t), dt + \sigma(X_t, t), dW_t, $$ where $W_t$ is a Brownian motion. Read it as a rule for taking a small step: over an interval $dt$, the state moves by a deterministic amount $\mu, dt$ (the drift) plus a random kick $\sigma, dW_t$ (the diffusion), where $dW_t$ is a Gaussian increment with mean zero and variance $dt$. The running example for this note is the Ornstein-Uhlenbeck (OU) process: $$ dX_t = -\theta X_t, dt + \sigma, dW_t. $$ The drift $-\theta X_t$ pulls the state back toward zero, more strongly the further out it is. The diffusion $\sigma, dW_t$ keeps kicking it away. This tug-of-war between mean reversion and noise is exactly the structure that shows up in interest-rate models, in the velocity of a particle under friction, and in many stationary time series. We will use $\theta = 1$, $\sigma = 1$, and start every path at $X_0 = 2$. The simplest way to simulate an SDE is the Euler-Maruyama scheme. Discretize time into steps of size $\Delta t$ and replace the differentials by increments: $$ X_{k+1} = X_k + \mu(X_k, t_k), \Delta t + \sigma(X_k, t_k), \sqrt{\Delta t}; Z_k, \qquad Z_k \sim \mathcal{N}(0, 1). $$ The only subtlety is that the random increment scales as $\sqrt{\Delta t}$, not $\Delta t$. That is the signature of Brownian motion: over a window of length $\Delta t$ the displacement has standard deviation $\sqrt{\Delta t}$, so variance grows linearly in time. import numpy as np def euler_maruyama(mu, sigma, x0, T, dt, n_paths, rng): n_steps = int(T / dt) X = np.full(n_paths, float(x0)) traj = np.empty((n_steps + 1, n_paths)) traj[0] = X for k in range(n_steps): t = k * dt Z = rng.standard_normal(n_paths) X = X + mu(X, t) * dt + sigma(X, t) * np.sqrt(dt) * Z traj[k + 1] = X return traj theta, sigma_ou, x0 = 1.0, 1.0, 2.0 mu = lambda x, t: -theta * x sig = lambda x, t: sigma_ou rng = np.random.default_rng(0) traj = euler_maruyama(mu, sig, x0, T=4.0, dt=1e-3, n_paths=20000, rng=rng)</code></pre> A handful of these paths, plotted over time, all start at $2$ and get dragged toward zero while jittering: Twelve OU sample paths. Each starts at X_0 = 2, is pulled toward zero by the drift, and jitters under the diffusion. No single path is predictable; the ensemble is.</figcaption> </figure> No individual path is predictable. But look at all $20{,}000$ of them at once and a pattern appears. From One Path to a Density</h2> Fix a time $t$ and forget the ordering of the paths. What we have is a cloud of values $X_t^{(1)}, X_t^{(2)}, \ldots$, one per path. Their histogram approximates a probability density $p(x, t)$: the probability of finding the process near $x$ at time $t$. At $t = 0$ that density is a spike at $x = 2$, because every path starts there. As time runs, two things happen at once. The whole cloud drifts left toward zero, following the mean reversion. And it spreads out, because the noise accumulates. Eventually the inward pull and the outward spreading balance, and the density stops changing. That limiting shape is the stationary distribution. The ensemble density at four times. It starts concentrated at X_0 = 2, then drifts toward zero and broadens, settling into a fixed bell curve. The Fokker-Planck equation is the law governing this whole movie.</figcaption> </figure> The question this note answers is: can we describe the movie of $p(x, t)$ directly, without simulating a single path? Yes. The density obeys a deterministic PDE, and that PDE is the Fokker-Planck equation. Deriving the Fokker-Planck Equation from Ito's Lemma</h2> The cleanest derivation uses a test function. Let $f$ be any smooth function that vanishes outside a bounded region. We track how the expectation $\mathbb{E}[f(X_t)]$ evolves, in two different ways, and match them. Route one: apply Ito's lemma. For $dX_t = \mu, dt + \sigma, dW_t$, Ito's lemma says that the differential of $f(X_t)$ carries an extra second-order term that ordinary calculus would miss: $$ df(X_t) = f'(X_t), dX_t + \tfrac{1}{2} f''(X_t), (dX_t)^2. $$ Using $(dX_t)^2 = \sigma^2, dt$ (the defining rule of Ito calculus, since $(dW_t)^2 = dt$), $$ df(X_t) = \left[\mu, f'(X_t) + \tfrac{1}{2}\sigma^2 f''(X_t)\right] dt + \sigma f'(X_t), dW_t. $$ Take expectations. The $dW_t$ term has mean zero, so it drops out: $$ \frac{d}{dt}, \mathbb{E}[f(X_t)] = \mathbb{E}!\left[\mu(X_t), f'(X_t) + \tfrac{1}{2}\sigma^2(X_t), f''(X_t)\right] = \int \left[\mu(x) f'(x) + \tfrac{1}{2}\sigma^2(x) f''(x)\right] p(x, t), dx. $$ Route two: differentiate under the integral. By definition $\mathbb{E}[f(X_t)] = \int f(x), p(x, t), dx$, so $$ \frac{d}{dt}, \mathbb{E}[f(X_t)] = \int f(x), \frac{\partial p}{\partial t}, dx. $$ Match them. Set the two expressions equal and move all derivatives off $f$ using integration by parts. Because $f$ vanishes at the boundaries, every boundary term is zero: $$ \int \mu f' p, dx = -\int f, \frac{\partial}{\partial x}(\mu p), dx, \qquad \int \tfrac{1}{2}\sigma^2 f'' p, dx = \int f, \frac{\partial^2}{\partial x^2}!\left(\tfrac{1}{2}\sigma^2 p\right) dx. $$ Substituting back, we get an identity that must hold for every test function $f$: $$ \int f(x), \frac{\partial p}{\partial t}, dx = \int f(x)\left[-\frac{\partial}{\partial x}(\mu p) + \frac{\partial^2}{\partial x^2}!\left(\tfrac{1}{2}\sigma^2 p\right)\right] dx. $$ If two integrals agree against all test functions, their integrands agree. This gives the Fokker-Planck equation (also called the forward Kolmogorov equation): $$ \boxed{;\frac{\partial p}{\partial t} = -\frac{\partial}{\partial x}\big[\mu(x, t), p\big] + \frac{\partial^2}{\partial x^2}!\left[\tfrac{1}{2}\sigma^2(x, t), p\right].;} $$ The derivation is worth pausing on. The first term came from the drift $\mu$ and is first order; the second came from the diffusion $\sigma$ and is second order, and it is there only because of the extra Ito term $\tfrac12 f''(dX)^2$. Without Ito calculus, the diffusion would vanish and the density would never spread. The whole "randomness" of the process lives in that one second-order term. What the Equation Actually Says: Transport Plus Spreading</h2> The Fokker-Planck equation is easier to read as a conservation law. Write it as $$ \frac{\partial p}{\partial t} = -\frac{\partial J}{\partial x}, \qquad J(x, t) = \mu(x, t), p - \frac{\partial}{\partial x}!\left[\tfrac{1}{2}\sigma^2(x, t), p\right]. $$ Here $J$ is the probability current: the rate at which probability mass flows past the point $x$. The equation $\partial_t p = -\partial_x J$ is just the statement that probability is conserved. If more flows in than out at a point, the density there rises. The current has two pieces, and they correspond exactly to the two forces in the SDE. Transport (drift). The term $\mu(x), p$ carries mass in the direction of the drift, like a wind blowing the density downstream. For the OU process $\mu = -\theta x$ blows everything toward the origin.</li> Spreading (diffusion). The term $-\partial_x[\tfrac12\sigma^2 p]$ pushes mass from high-density regions to low-density ones, exactly like heat flowing from hot to cold. It is what turns a spike into a bell curve.</li> </ul> A stationary distribution $p_\infty$ is one where the movie stops: $\partial_t p = 0$, which means the current $J$ is constant. On the whole line, with density decaying to zero at infinity, that constant must be zero. Setting $J = 0$ for the OU process, $$ -\theta x, p_\infty - \frac{\sigma^2}{2}, p_\infty' = 0 \quad\Longrightarrow\quad \frac{p_\infty'}{p_\infty} = -\frac{2\theta}{\sigma^2}, x \quad\Longrightarrow\quad p_\infty(x) \propto \exp!\left(-\frac{\theta x^2}{\sigma^2}\right). $$ That is a Gaussian with mean $0$ and variance $\sigma^2 / (2\theta)$. Drift and diffusion reach a truce, and the truce is a bell curve. With our numbers $\theta = \sigma = 1$, the stationary law is $\mathcal{N}(0, \tfrac12)$. The Ornstein-Uhlenbeck Answer in Closed Form</h2> The OU process is special: we can solve it exactly, which makes it the perfect test case. Because the drift is linear and the noise is additive, the transition density stays Gaussian for all time. Starting from a fixed $X_0 = x_0$, the process at time $t$ is normally distributed: $$ X_t \sim \mathcal{N}\big(m(t),, v(t)\big), $$ $$ m(t) = x_0, e^{-\theta t}, \qquad v(t) = \frac{\sigma^2}{2\theta}\left(1 - e^{-2\theta t}\right). $$ The mean decays exponentially toward zero at rate $\theta$. The variance grows from $0$ up to the stationary value $\sigma^2 / (2\theta)$. Both limits match the picture: the cloud drifts to the origin and its width saturates. You can verify these solve the Fokker-Planck equation by substitution, but it is more convincing to check them numerically, which is what the rest of the note does. Hand Calculation with Real Numbers</h2> Before writing any solver, pin down what the answer should be at a specific time, so we have something concrete to test against. Take $t = 1$ with $\theta = \sigma = 1$ and $x_0 = 2$. Mean. $$ m(1) = 2, e^{-1} = 2 \times 0.367879 = 0.735759. $$ Variance. $$ v(1) = \frac{1}{2}\left(1 - e^{-2}\right) = \frac{1}{2}\left(1 - 0.135335\right) = \frac{1}{2}\times 0.864665 = 0.432332, $$ so the standard deviation is $\sqrt{0.432332} = 0.657520$. Stationary check. As $t \to \infty$, $m(t) \to 0$ and $v(t) \to \tfrac12$, matching the $\mathcal{N}(0, \tfrac12)$ we derived from setting the current to zero. Its standard deviation is $\sqrt{0.5} = 0.707107$. So at $t = 1$ the density should be a Gaussian centered at about $0.736$ with standard deviation about $0.658$, and by $t = 4$ it should be almost exactly $\mathcal{N}(0, \tfrac12)$. Keep these three numbers in mind; every method below has to reproduce them. Solving the Fokker-Planck Equation from Scratch</h2> We now solve the PDE numerically, without simulating any paths. Discretize $x$ on a grid and integrate the conservation law forward in time. The safe way to do this is to keep the flux form, so that the scheme conserves total probability by construction: compute the current $J$ on the cell faces (the midpoints between grid points), then update each cell by the difference of the fluxes on its two faces. def solve_fokker_planck(mu, D, x, p0, T, dt): """Explicit finite-volume solver for dp/dt = -d/dx(mu p) + d^2/dx^2(D p). mu(x): drift, D: constant diffusion sigma^2/2, x: grid, p0: initial density.""" dx = x[1] - x[0] xf = 0.5 * (x[:-1] + x[1:]) # cell faces (midpoints) muf = mu(xf) p = p0.copy() n_steps = int(T / dt) for _ in range(n_steps): # probability current on the faces: transport minus diffusion J = muf * 0.5 * (p[:-1] + p[1:]) - D * (p[1:] - p[:-1]) / dx dpdt = np.zeros_like(p) dpdt[1:-1] = -(J[1:] - J[:-1]) / dx # divergence of the current p = p + dt * dpdt p[0] = p[-1] = 0.0 # absorbing far-field boundary return p # grid and a narrow-Gaussian approximation to the initial spike at x0 x = np.linspace(-4, 4, 801) dx = x[1] - x[0] p0 = np.exp(-(x - x0) ** 2 / (2 * 0.01)) / np.sqrt(2 * np.pi * 0.01) D = sigma_ou ** 2 / 2 dt_pde = 0.2 * dx ** 2 / D # diffusion stability limit (CFL) p_pde = solve_fokker_planck(lambda z: -theta * z, D, x, p0, T=1.0, dt=dt_pde)</code></pre> Two numerical points matter. First, the time step is bounded by a stability condition: for an explicit diffusion scheme, $\Delta t \lesssim \Delta x^2 / \sigma^2$. Push past it and the solution blows up into oscillations. Second, the initial spike at $x_0$ is approximated by a narrow Gaussian, because a true delta on a grid is a single spike that the scheme struggles with; the diffusion smooths this out almost immediately, so it does not affect the answer at $t = 1$. Three Routes, One Answer</h2> Now the payoff. We have three completely independent ways to get the density at $t = 1$: Monte Carlo. Simulate many SDE paths with Euler-Maruyama and histogram their values at $t = 1$.</li> PDE. Solve the Fokker-Planck equation on a grid and read off $p(x, 1)$.</li> Closed form. Evaluate the exact Gaussian $\mathcal{N}(m(1), v(1))$.</li> </ol> from math import erf # only for the analytic pdf below # 1. Monte Carlo: reuse the trajectories, take the slice at t = 1 t_idx = int(1.0 / 1e-3) mc_samples = traj[t_idx] # 3. Closed-form transition density at t = 1 m1 = x0 * np.exp(-theta * 1.0) v1 = (sigma_ou ** 2 / (2 * theta)) * (1 - np.exp(-2 * theta * 1.0)) analytic = np.exp(-(x - m1) ** 2 / (2 * v1)) / np.sqrt(2 * np.pi * v1) print(f"analytic mean={m1:.6f} std={np.sqrt(v1):.6f}") print(f"monte carlo mean={mc_samples.mean():.6f} std={mc_samples.std():.6f}") # PDE mean/std from the grid density p_pde print(f"pde grid mean={np.sum(x * p_pde) * dx:.6f} " f"std={np.sqrt(np.sum((x - np.sum(x * p_pde) * dx) ** 2 * p_pde) * dx):.6f}")</code></pre> All three report the same numbers we computed by hand, up to discretization error: analytic mean=0.735759 std=0.657520 monte carlo mean=0.738001 std=0.656903 pde grid mean=0.735715 std=0.658482</code></pre> Overlaid on the same axes, the Monte Carlo histogram, the PDE solution, and the analytic Gaussian sit on top of each other: The density at t = 1, computed three independent ways. The Monte Carlo histogram (bars), the Fokker-Planck PDE solution (solid), and the closed-form Gaussian (dashed) agree because they describe the same object from different directions.</figcaption> </figure> This is the moment where the theory becomes tangible. The Monte Carlo route never writes down a PDE; it just pushes paths forward and counts. The PDE route never simulates a path; it evolves a density on a grid. The closed form does neither; it solves the equation exactly. Three different arithmetic routes, one distribution, because the Fokker-Planck equation is precisely the deterministic law that the random ensemble obeys. The Stationary Distribution</h2> Let the same three methods run out to $t = 4$ and the density settles onto the stationary law $\mathcal{N}(0, \tfrac12)$ we derived by setting the probability current to zero: By t = 4 the ensemble has forgotten where it started. The long-time density (bars and solid) matches the stationary Gaussian N(0, 1/2) (dashed), the balance point where drift and diffusion cancel.</figcaption> </figure> The process has forgotten its initial condition. Whether it started at $x_0 = 2$ or anywhere else, it relaxes to the same bell curve. That loss of memory, at rate $\theta$, is the defining property of a mean-reverting process, and the Fokker-Planck equation makes it quantitative. Practical Points Worth Making Explicit</h2> A few things trip people up when they take this beyond the OU example. Stability of the explicit scheme. The time step must respect $\Delta t \lesssim \Delta x^2 / \sigma^2$. Halving the grid spacing quarters the allowed step. When this becomes expensive, implicit schemes (Crank-Nicolson) remove the restriction at the cost of solving a linear system each step. Boundary conditions carry modelling meaning. We used an absorbing far-field boundary ($p = 0$), which is fine when the density decays anyway. But boundaries are not just numerics: a reflecting boundary (zero current, $J = 0$) models a wall the process cannot cross, while an absorbing boundary models a level at which the process is killed. In finance the absorbing case is exactly a barrier option or a default threshold. State-dependent diffusion. When $\sigma$ depends on $x$, the second term is $\partial_x^2[\tfrac12 \sigma^2(x), p]$, and the $\sigma^2$ stays inside both derivatives. Geometric Brownian motion, $dX = \mu X, dt + \sigma X, dW$, is the standard example and underlies Black-Scholes. Getting that placement wrong is the most common error in writing down a Fokker-Planck equation. Higher dimensions. In $d$ dimensions the drift becomes a vector and the diffusion a matrix $D = \tfrac12 \sigma \sigma^\top$, and the equation reads $\partial_t p = -\nabla!\cdot(\boldsymbol\mu p) + \nabla!\cdot(\nabla!\cdot(D, p))$. The grid solver stops scaling, and Monte Carlo becomes the practical method. That crossover is why simulation dominates high-dimensional problems. The Big Picture</h2> At this point the pieces should feel like one continuous argument rather than a definition dropped from the sky. We started with an SDE, a rule for one random trajectory, and simulated it with Euler-Maruyama. One path is unpredictable, but the ensemble of paths forms a density $p(x, t)$. Applying Ito's lemma to a test function and integrating by parts turned the randomness of the SDE into a deterministic PDE for that density, the Fokker-Planck equation: $$ \frac{\partial p}{\partial t} = -\frac{\partial}{\partial x}\big[\mu, p\big] + \frac{\partial^2}{\partial x^2}!\left[\tfrac{1}{2}\sigma^2, p\right]. $$ Reading it as a conservation law $\partial_t p = -\partial_x J$ exposed its two forces: drift transports probability, diffusion spreads it. Setting the current to zero gave the stationary distribution. And an Ornstein-Uhlenbeck example, solvable in closed form, let us check three independent routes, Monte Carlo, a from-scratch PDE solver, and the exact Gaussian, against the same hand-computed numbers. So the SDE and the Fokker-Planck equation are two views of one process. The SDE is the microscopic rule for a single sample; the Fokker-Planck equation is the macroscopic law for the whole probability cloud. This is the same duality that runs through the rest of the stochastic toolkit: the forward Kolmogorov equation here evolves densities in time, while the backward Kolmogorov equation evolves expectations and is the object behind Feynman-Kac and option pricing. I think this topic reads better when the SDE is taken as the starting point and the PDE is derived as its consequence, rather than the other way around. The mathematics carries the weight, the OU example makes the abstraction concrete, and the three-way agreement closes the loop. If you think there is a better example, a cleaner derivation, or a natural extension, feel free to suggest it. From here the obvious next steps are the backward Kolmogorov equation and Feynman-Kac, Langevin dynamics as a sampler, and the jump from these scalar equations to their high-dimensional, matrix-diffusion form.

An Introduction to MLOps: A Complete Tutorial with a Hands-On

2026-06-29T00:00:00+00:00

This is the written companion to the Introduction to MLOps slides</a>. It follows the deck section by section: the same chain of ideas, the same figures, and then the hands-on the deck builds toward. The slides give the picture in a room; this note lets you read it at your own pace and, more importantly, run it. By the end you will have tracked a real machine-learning experiment, compared two runs, and inspected them in a web UI.
The chain the deck walks, and so will we:

reproducibility → the software cycle → DevOps → the ML lifecycle → MLOps → MLflow → a hands-on. </blockquote>
In a hurry? Skip straight to the hands-on</a>, or just git clone</code> the companion repo, github.com/JienWeng/mlops-tutorial</a>, and run it. The sections above the hands-on are the why; the hands-on is the how. </blockquote>
Why reproducibility, and why ML makes it hard</h2> The starting motivation is research integrity. A result you cannot reproduce is hard to trust, hard to build on, and hard to defend. In software-heavy science this is why people publish code alongside papers (the Papers with Code</a> culture). But "publish the code" is not enough on its own. Sharing code with papers is a start. Credit: paperswithcode.com</figcaption> </figure> Reproducibility really means being able to recreate a result from all of its ingredients: the code (which commit?),</li> the data (which version, which split?),</li> the environment (which library versions?),</li> and the parameters that produced the run.</li> </ul> What reproducibility really implies: far more than just the code.</figcaption> </figure> Machine learning makes this harder than ordinary software because three things change over time, not one: Artifact</th> Changes because…</th></tr></thead> Code</td> the usual reasons: refactors, bug fixes, new features</td></tr> Data</td> new data arrives, distributions drift, labels get corrected</td></tr> Model</td> it is retrained, fine-tuned, or replaced</td></tr> </tbody></table> Track only the code and you will still fail to reproduce a model, because the data and the trained model moved underneath you. MLOps is the practice of bringing software-engineering discipline (automation, reproducibility, monitoring) to all three. The one-line definition worth memorising: MLOps = DevOps principles applied to machine-learning systems. </blockquote> To get there, we first need the software-engineering half of that sentence. The software development cycle</h2> Classic software development moves through repeating stages: Development: planning and coding.</li> Integration: quick tests and a build.</li> Testing: deeper tests, validation, and a release.</li> Delivery: final packaging and deployment to a running server.</li> Monitoring: collecting data, watching each component, gathering user feedback.</li> </ol> Run those once, by hand, and you have traditional software delivery: slow, batched, and error-prone at the hand-offs. The stages of the software development cycle.</figcaption> </figure> Automating it: DevOps and CI/CD</h2> DevOps is what you get when you automate that cycle so every change flows through it continuously instead of in big manual batches: Continuous Integration (CI): every commit is automatically built and tested.</li> Continuous Delivery (CD): passing builds are automatically packaged and deployed.</li> Continuous Monitoring: the running system is watched for errors and feedback, which flows back into planning.</li> </ul> CI/CD automates the loop: every change is built, tested, and shipped continuously. Credit: resources.github.com/ci-cd</figcaption> </figure> DevOps is usually described as resting on three legs, often drawn as a trident: People: a culture where development and operations share ownership.</li> Process: the automated CI/CD pipeline itself.</li> Tools: the machinery that runs it (GitHub Actions, GitLab CI, containers, and so on).</li> </ul> The tools get the attention, but the culture and the process are what make the automation stick. DevOps rests on three legs: people, process, and tools. Credit: aws.amazon.com/devops</figcaption> </figure> The machine-learning lifecycle</h2> Machine learning adds its own stages on top of the software cycle. A typical ML lifecycle runs: Data extraction: fetch the data.</li> Data analysis: understand its nature and quirks.</li> Data preparation: clean it, engineer features, split into train/validation/test.</li> Model training: fit the model, tune hyperparameters.</li> Model evaluation: measure quality on held-out data.</li> Model validation: confirm it beats a baseline and is fit to deploy.</li> Model serving: package and deploy it to make predictions.</li> Model monitoring: watch performance and decide when to retrain.</li> </ol> The machine-learning cycle. Credit: ml-ops.org</figcaption> </figure> The same lifecycle, end to end: data stages feed model stages, and monitoring loops back to the start.</figcaption> </figure> The loop at the end is the important part: monitoring feeds back into data and training. An ML system in production is a living thing, not a shipped artifact. MLOps = DevOps for ML</h2> Now overlay the DevOps loop onto the ML lifecycle. Each "continuous" practice gains an ML twist: MLOps = DevOps principles applied to ML systems. Credit: ml-ops.org</figcaption> </figure> Continuous Integration is no longer only about testing code: it now also tests and validates data, schemas, and models.</li> Continuous Delivery ships not a single package but a whole pipeline that can deploy a model-serving service.</li> Continuous Training (CT) is unique to ML: the system can automatically retrain and redeploy as new data arrives.</li> Continuous Monitoring tracks model decay and can trigger retraining when quality drops.</li> </ul> CT and model/data monitoring are what make MLOps its own discipline rather than "DevOps with notebooks". What an MLOps workflow looks like in practice</h2> Put concretely, a working MLOps setup wires together a handful of pieces: code versioning (git) and data versioning (DVC): the two moving inputs, tracked like any other asset;</li> experimentation in notebooks (Jupyter), feeding an experiment-tracking server (MLflow) that records every run's parameters and metrics;</li> artifact tracking (also MLflow): every file a run produces (the trained model, a confusion-matrix plot, a preprocessing pipeline) gets stored and linked to the run that made it;</li> a model registry (MLflow) that versions the best models and stages them (staging → production);</li> pipeline orchestration (Airflow) that wires the above into a scheduled, repeatable flow;</li> model serving (BentoML) that wraps a registered model in an API; and</li> model monitoring (Prometheus/Grafana) that watches the served model and closes the loop back to data.</li> </ul> A concrete MLOps workflow wiring the pieces together. Credit: mymlops.com</figcaption> </figure> Experiment tracking and artifact tracking are easy to conflate; MLflow happens to do both, which is why it appears three times in the diagram. The distinction matters: experiment tracking logs the numbers (mlflow.log_metric("accuracy", acc)</code>), while artifact tracking stores the files a run produced (mlflow.log_artifact("confusion_matrix.png")</code>). Same tool, two different jobs. You will use both in the hands-on below. You do not build all of this on day one. The single highest-leverage habit, the one that pays off immediately even in a solo project, is experiment tracking. That is where we will get hands-on, before bridging to model serving at the end. MLflow: the tool we will use</h2> MLflow</a> is the open-source standard for experiment tracking. The mental model: an ML experiment is environment + data + code, and MLflow is the logbook that records, for each run, its hyperparameters + results + plots so you can compare runs later in a web UI. It is framework-agnostic (scikit-learn, PyTorch, TensorFlow, XGBoost…) and works for any computational experiment, not just ML. MLflow has four components: MLflow's components. Credit: mlflow.org</figcaption> </figure> How the pieces fit together when you track an experiment. Credit: mlflow.org</figcaption> </figure> Tracking: log parameters, metrics, and artifacts per run (our focus).</li> Projects: package code so a run is reproducible.</li> Models: a standard packaging format so a trained model can be served anywhere.</li> Model Registry: version models and move them through stages (staging → production).</li> </ul> The MLflow tracking API</h2> Tracking comes down to a few calls. First, name an experiment to group related runs, then wrap each run in a context manager: import mlflow mlflow.set_experiment("intro-to-mlops") with mlflow.start_run(run_name="my-first-run"): mlflow.log_param("learning_rate", 0.01) # one hyperparameter mlflow.log_params({"epochs": 20, "batch": 32}) # several at once mlflow.log_metric("accuracy", 0.92) # a result mlflow.sklearn.log_model(model, name="model") # the model itself</code></pre> The three verbs you will use constantly: log_param</code> / log_params</code>: key-value settings (hyperparameters, config).</li> log_metric</code>: numeric results (loss, accuracy); these become plots in the UI so you can compare runs.</li> log_model</code>: the trained model, in MLflow's portable format.</li> </ul> Too lazy to log by hand? For supported libraries, a single autolog()</code> call before fit()</code> captures parameters, metrics, and the model automatically: mlflow.sklearn.autolog() # or mlflow.tensorflow.autolog(), mlflow.pytorch.autolog() model.fit(X_train, y_train) # params, metrics, and model are logged for you</code></pre> Manual logging gives you the most control; autolog</code> gives you most of the value for one line. And the feature people quietly love most: logging plots as artifacts, so a figure is stored with the run and viewable in the UI: plt.savefig("loss.png") mlflow.log_artifact("loss.png")</code></pre>Where MLflow stores runs: three setup options</h2> Before logging anything, MLflow needs to know where runs go. There are three options, in increasing order of scale: MLflow tracking setup. We use the local-database option below. Credit: mlflow.org</figcaption> </figure> Option 1, local filesystem (./mlruns</code>): the simplest default.</li> Option 2, local database (sqlite:///mlflow.db</code>): still on your laptop, but the backend MLflow now recommends. This is what we use below.</li> Option 3, remote server (http://server:5000</code>): a shared tracking server for a team.</li> </ul> Hands-on: track your first experiment</h2> Everything below is self-contained and runs on a laptop in under ten minutes. We will train two small classifiers, log them to MLflow, and compare them in the UI. Prerequisites: Python 3.9+ and pip</code>. No GPU, no cloud account, no prior MLflow setup. Five steps: install → write the script → run it → run it again → open the UI. Want to just clone and run? The full source (script and notebook) lives in the companion repo: github.com/JienWeng/mlops-tutorial</a>. git clone</code> it, pip install -r requirements.txt</code>, then python train.py</code> (or open notebook.ipynb</code>). The steps below explain what that code does. </blockquote> 1. Install</h3> python -m pip install mlflow scikit-learn matplotlib</code></pre>2. Choose a store (Option 2: local SQLite)</h3> We use a local SQLite database, Option 2 above. It keeps everything on your laptop with no server to run. Heads-up (MLflow 3.x): the plain-folder store (./mlruns</code> on its own) is now in maintenance mode and will raise an error unless you opt in. A SQLite URI like sqlite:///mlflow.db</code> is the friction-free local setup, so that is what we use. (On MLflow 2.x this same code still works.) </blockquote> We set the store directly in the script, so there is nothing to configure separately. 3. The experiment script</h3> Save this as train.py</code>. It trains a logistic-regression classifier on the classic breast-cancer dataset, then logs the parameters, the metrics, the trained model, and a confusion-matrix plot. # train.py import matplotlib.pyplot as plt import mlflow import mlflow.sklearn from sklearn.datasets import load_breast_cancer from sklearn.linear_model import LogisticRegression from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score, f1_score from sklearn.model_selection import train_test_split # Where to store runs: a local SQLite db (metadata) + ./mlruns (artifacts) mlflow.set_tracking_uri("sqlite:///mlflow.db") # Group related runs under a named experiment mlflow.set_experiment("intro-to-mlops") # --- data --- X, y = load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # A hyperparameter we want to track and vary later C = 1.0 # inverse regularisation strength with mlflow.start_run(run_name=f"logreg-C={C}"): # --- train --- model = LogisticRegression(C=C, max_iter=10_000) model.fit(X_train, y_train) # --- evaluate --- preds = model.predict(X_test) acc = accuracy_score(y_test, preds) f1 = f1_score(y_test, preds) # --- log: parameters, metrics, the model itself --- mlflow.log_param("model", "LogisticRegression") mlflow.log_param("C", C) mlflow.log_metric("accuracy", acc) mlflow.log_metric("f1", f1) mlflow.sklearn.log_model(model, name="model") # --- log: a plot as an artifact --- ConfusionMatrixDisplay.from_predictions(y_test, preds) plt.title(f"Confusion matrix (C={C})") plt.savefig("confusion_matrix.png", bbox_inches="tight") mlflow.log_artifact("confusion_matrix.png") print(f"Logged run: accuracy={acc:.4f}, f1={f1:.4f}")</code></pre> Run it: python train.py</code></pre> You just produced your first tracked run. The parameters, metrics, the serialised model, and the plot are all saved. 4. Make a second run to compare against</h3> Change one line, C = 0.01</code>, and run python train.py</code> again. (Stronger regularisation; the score should move.) Now you have two runs to compare, which is the whole point of tracking. 5. Inspect and compare in the UI</h3> From the same folder, start the UI, pointing it at the same SQLite store: mlflow ui --backend-store-uri sqlite:///mlflow.db</code></pre> Open http://127.0.0.1:5000</a>. You will see the intro-to-mlops</code> experiment with both runs. It looks like this: The MLflow UI: every run with its parameters and metrics, ready to sort and compare. Credit: mlflow.org</figcaption> </figure> From here you can: sort and filter runs by accuracy</code> or f1</code>,</li> tick two runs and click Compare to see parameters and metrics side by side,</li> open a run to view its confusion-matrix plot and download the saved model.</li> </ul> That comparison view answers the question MLOps exists to answer: which settings gave the best score, and exactly how were they produced? You now have a reproducible record of it. 6. Serve it: MLflow → BentoML (minimal)</h3> Tracking answers "what's the best run?" Serving answers "how do users call it?" This is the model serving box from the workflow diagram above, and BentoML</a> is the tool. The bridge is one call: load the model MLflow already tracked and stored, hand it to a Bento service. python -m pip install bentoml</code></pre> Save this as service.py</code>, next to mlflow.db</code> and mlruns/</code> (also in the companion repo, alongside train.py</code>): # service.py import bentoml import mlflow mlflow.set_tracking_uri("sqlite:///mlflow.db") # Grab the most recent run of the experiment, no run ID to copy by hand experiment = mlflow.get_experiment_by_name("intro-to-mlops") latest_run = mlflow.search_runs( experiment.experiment_id, order_by=["start_time DESC"], max_results=1 ).iloc[0] model = mlflow.sklearn.load_model(f"runs:/{latest_run.run_id}/model") @bentoml.service class MLopsService: @bentoml.api def predict(self, features: list[list[float]]) -> list[int]: return model.predict(features).tolist()</code></pre> Run it: bentoml serve service:MLopsService</code></pre> That starts an HTTP API on http://127.0.0.1:3000</a> with interactive docs. Call it: curl -X POST http://127.0.0.1:3000/predict \ -H "Content-Type: application/json" \ -d '{"features": [[14.0, 20.0, 90.0, 600.0, 0.1, 0.1, 0.1, 0.05, 0.2, 0.06, 0.4, 1.0, 3.0, 40.0, 0.006, 0.02, 0.03, 0.01, 0.02, 0.003, 16.0, 25.0, 105.0, 900.0, 0.14, 0.25, 0.3, 0.12, 0.3, 0.08]]}'</code></pre> That's the whole bridge: MLflow tracks and stores the model; BentoML loads it by URI and serves it. Everything past this (batching, Docker packaging with bentoml build</code>, autoscaling) is the same idea scaled up. What you just built (and what comes next)</h3> You have implemented the smallest real MLOps loop: parameterise → run → log → compare → serve. The natural extensions, in rough order of payoff: Commit train.py</code> to git so each run is tied to a code version.</li> Register the best model in MLflow's Model Registry to give it a name and version, instead of pasting a run ID.</li> Automate the run in CI so a push retrains and re-logs.</li> Monitor the served model's predictions, closing the loop back to data.</li> </ol> That list is exactly the MLOps workflow from earlier. You have now built its first two links. Take-home message</h2> Research, and production work too, should be reproducible and, where possible, open. If your work involves machine learning, that means using dedicated tools to make experiments reproducible rather than relying on memory and scattered notebooks. Adopting these habits in your daily workflow pays off quickly: time saved in the long run, engineering skill gained, and trust earned among collaborators. And MLflow in particular is not just for machine learning: it is a capable logbook for tracking any computational experiment and keeping your results reproducible over time. .small[This tutorial follows and is adapted from the lecture "An introduction to MLOps" by Alexandre Boucaud</a> (LSST France, Lyon, December 2023), licensed under CC BY-SA 4.0</a>. The figures are taken from the accompanying slides</a>, which are re-hosted under the same license; individual figures credit their original sources (ml-ops.org, mlflow.org, mymlops.com, GitHub, AWS) in their captions. This written tutorial and hands-on are an original adaptation, likewise shared under CC BY-SA 4.0.] Tree-Based Models Explained: From Decision Trees to Random Forest and XGBoost 2026-05-11T00:00:00+00:00 A single decision tree is powerful but unstable. Nearly everything about Random Forests and XGBoost follows from that one sentence, so that is where this note starts, not at the formulas. Forests and boosting are just two different answers to the instability problem. The path is: $$ \text{decision rules} \to \text{decision trees} \to \text{bagging} \to \text{Random Forest} \to \text{boosting} \to \text{XGBoost}. $$ Intuition first, but the math is written out in full. I dislike formulas that appear from nowhere. Run it yourself. The worked example is collected in a runnable notebook: download tree-based-models.ipynb</code></a>. It uses numpy</code>, scikit-learn</code>, and (optionally) xgboost</code>. </blockquote> Why People Reach for Trees, Random Forests, and XGBoost</h2> The motivation usually looks like this: Linear models miss nonlinear structure. Trees capture nonlinear, rule-like boundaries without feature engineering.</li> Single trees are unstable. They are accurate but high-variance.</li> Random Forests stabilize trees. Bagging reduces variance by averaging many noisy trees.</li> Boosting repairs bias. XGBoost builds trees sequentially to correct systematic mistakes.</li> </ul> So the choice is not "trees vs. XGBoost". It is more like this: Use a single tree to explain a small, interpretable rule set.</li> Use a Random Forest when you want strong performance with low tuning risk.</li> Use XGBoost when you want maximum accuracy and can tune regularization.</li> </ul> Start with the Simplest Useful Model: A Decision Rule</h2> Imagine a classifier that uses a single rule: $$ \text{if } x_1 \le 3.5, \text{ predict class 0, else class 1}. $$ This is a decision stump. It is weak, but it is easy to interpret. The point is not accuracy. The point is that rules like this are already nonlinear in the raw feature space. Now chain many rules in sequence, and you get a decision tree. A single decision rule creates a nonlinear boundary from a linear threshold.</figcaption> </figure> Decision Trees as Greedy Partitioning</h2> A decision tree repeatedly splits the feature space into regions. Each split chooses a feature and a threshold. The goal is to make the resulting child nodes more "pure" than the parent node. For classification, the standard impurity measures are entropy and Gini. Entropy for a node with class proportions $p_k$ is $$ H(S) = -\sum_{k=1}^{K} p_k \log p_k. $$ Gini impurity is $$ G(S) = 1 - \sum_{k=1}^{K} p_k^2. $$ Both measures are $0$ when a node is perfectly pure and larger when it is mixed. Information Gain</h3> Suppose a candidate split divides a node $S$ into children $S_L$ and $S_R$. Let $n$ be the total samples in $S$, with $n_L$ and $n_R$ in the children. The information gain is $$ \text{IG}(S, \text{split}) = H(S) - \frac{n_L}{n} H(S_L) - \frac{n_R}{n} H(S_R). $$ Using Gini, replace $H$ with $G$ in the same formula. The tree algorithm is greedy: at each node it chooses the split that maximizes this gain. A Small Example</h3> Assume a node has $10$ samples: $6$ of class 1 and $4$ of class 0. The entropy is $$ H(S) = -\left(\frac{6}{10}\log\frac{6}{10} + \frac{4}{10}\log\frac{4}{10}\right). $$ Now imagine a split that creates: Left child: $4$ class 1, $1$ class 0 ($n_L=5$)</li> Right child: $2$ class 1, $3$ class 0 ($n_R=5$)</li> </ul> Compute $H(S_L)$ and $H(S_R)$, and you get a positive information gain. The split is not perfect, but it is better than the parent. That is enough to move the algorithm forward. A split is chosen because it reduces impurity, not because it is globally optimal.</figcaption> </figure> Leaf Predictions</h3> Once a node stops splitting, it becomes a leaf. For classification, the leaf prediction is $$ \hat{p}_k = \frac{\text{count of class } k \text{ in leaf}}{\text{samples in leaf}}, $$ and the predicted class is the argmax of $\hat{p}_k$. For regression, the leaf prediction is typically the mean: $$ \hat{y} = \frac{1}{|S|} \sum_{i \in S} y_i. $$ Why Trees Are Unstable</h3> A tree is a high-variance model. A small change in the dataset can change an early split, which changes every downstream region. Trees can overfit without strong regularization such as max depth, minimum samples per leaf, or pruning. This is where ensembles enter the story. Bagging: Reduce Variance by Averaging</h2> Bagging (bootstrap aggregating) trains many trees on different bootstrap samples of the data and averages their predictions. Let $\hat{f}_1, \dots, \hat{f}_M$ be $M$ trees. The bagged predictor is $$ \hat{f}_{\text{bag}}(x) = \frac{1}{M} \sum^M \hat{f}_m(x). $$ If each tree has variance $\sigma^2$ and pairwise correlation $\rho$, then $$ \text{Var}(\hat{f}_{\text{bag}}) = \frac{1}{M}(1-\rho)\sigma^2 + \rho\sigma^2. $$ So averaging reduces variance, but correlation limits how much we gain. Random Forest: Bagging + Feature Randomness</h2> Random Forests go one step further. They reduce correlation by forcing each split to consider only a random subset of features. This does two things: It decorrelates the trees, which reduces the $\rho$ term above.</li> It prevents strong predictors from dominating every split.</li> </ol> The forest prediction is still an average (regression) or a majority vote (classification). The math is the same as bagging, but the diversity is higher. Random Forests reduce variance by averaging many decorrelated trees.</figcaption> </figure> Boosting: Reduce Bias by Building Sequentially</h2> Bagging attacks variance. Boosting attacks bias. The idea is to build an additive model that corrects its own mistakes: $$ \hat{y}^{(t)}(x) = \hat{y}^{(t-1)}(x) + f_t(x), $$ where each $f_t$ is a small tree and the sequence is trained to reduce a loss function. Boosting is best understood as functional gradient descent: we are minimizing a loss by moving in function space, one tree at a time. XGBoost: Boosting with a Second-Order Objective</h2> XGBoost formalizes this with an objective that includes regularization. At step $t$: $$ \mathcal{L}^{(t)} = \sum_{i=1}^n l\left(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)\right) + \Omega(f_t), $$ where $l$ is the training loss and the regularization term is $$ \Omega(f) = \gamma T + \frac{1}{2}\lambda \sum_{j=1}^{T} w_j^2. $$ Here $T$ is the number of leaves and $w_j$ is the score assigned to leaf $j$. Second-Order Approximation</h3> Let $$ g_i = \partial_{\hat{y}} l(y_i, \hat{y}_i^{(t-1)}), $$ $$ h_i = \partial_{\hat{y}}^2 l(y_i, \hat{y}_i^{(t-1)}). $$ Using a second-order Taylor expansion around $\hat{y}_i^{(t-1)}$, the objective becomes $$ \tilde{\mathcal{L}}^{(t)} = \sum_{i=1}^n \left[g_i f_t(x_i) + \frac{1}{2} h_i f_t(x_i)^2\right] + \Omega(f_t). $$ Because a tree predicts a constant weight $w_j$ for all samples in leaf $j$, we can group terms per leaf. Let $I_j$ be the indices in leaf $j$ and define $$ G_j = \sum_{i \in I_j} g_i, \qquad H_j = \sum_{i \in I_j} h_i. $$ Then the optimal leaf weight is $$ w_j^* = -\frac{G_j}{H_j + \lambda}, $$ and the best value of the objective for that leaf is $$ -\frac{1}{2} \frac{G_j^2}{H_j + \lambda}. $$ Split Gain</h3> A split is good if it improves the objective. The gain of splitting a node into left and right children is $$ \text{Gain} = \frac{1}{2}\left(\frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{G^2}{H + \lambda}\right) - \gamma. $$ This is the exact reason XGBoost splits where it does. The split is not only about purity. It is about how much the second-order objective improves after regularization. XGBoost chooses splits by maximizing second-order gain with regularization.</figcaption> </figure> Why XGBoost Works So Well</h2> The success of XGBoost is not a mystery. It is the combination of: Sequential correction: each tree targets the remaining error.</li> Second-order curvature: Hessians stabilize the optimization.</li> Regularization: the $\gamma$ and $\lambda$ terms actively penalize complexity.</li> Shrinkage: a learning rate scales each tree so the model does not overreact.</li> </ul> In practice, these ideas mean the model stays expressive without exploding in variance. Boosting builds an additive model one small tree at a time.</figcaption> </figure> A Few Practical Tradeoffs</h2> Tree ensembles are strong, but their strengths are specific: Interpretability: single trees are readable, forests and boosted trees are not.</li> Bias vs variance: Random Forests reduce variance, boosting reduces bias.</li> Overfitting: deep trees can overfit, especially in boosting without regularization.</li> Data leakage: trees can memorize target leakage quickly, so features must be clean.</li> Class imbalance: impurity is dominated by the majority class unless weighted.</li> Compute cost: XGBoost is fast for trees, but still heavier than linear models.</li> </ul> A Worked Dataset Example (Synthetic Regression)</h2> To make this concrete, we use a small synthetic regression dataset from scikit-learn. It is numeric, quick to run, and avoids any external downloads. The idea is to compare a single tree, a Random Forest, and XGBoost on the same train/test split. A synthetic regression dataset with noise and nonlinear structure.</figcaption> </figure> Below is a compact code snippet. The full notebook</a> is provided separately so you can run it end-to-end. import numpy as np from sklearn.datasets import make_regression from sklearn.model_selection import train_test_split from sklearn.metrics import mean_absolute_error, mean_squared_error from sklearn.tree import DecisionTreeRegressor from sklearn.ensemble import RandomForestRegressor try: from xgboost import XGBRegressor except ImportError: XGBRegressor = None X, y = make_regression(n_samples=2000, n_features=12, noise=15.0, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) tree = DecisionTreeRegressor(max_depth=4, random_state=42) rf = RandomForestRegressor(n_estimators=300, random_state=42) models = {"tree": tree, "rf": rf} if XGBRegressor is not None: xgb = XGBRegressor( n_estimators=400, max_depth=4, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, reg_lambda=1.0, random_state=42, ) models["xgb"] = xgb for name, model in models.items(): model.fit(X_train, y_train) preds = model.predict(X_test) rmse = np.sqrt(mean_squared_error(y_test, preds)) mae = mean_absolute_error(y_test, preds) print(name, "rmse", round(rmse, 3), "mae", round(mae, 3))</code></pre> RMSE and MAE comparison across a single tree, Random Forest, and XGBoost.</figcaption> </figure> Feature importance comparison for Random Forest and XGBoost.</figcaption> </figure> What Code to Look At</h2> If you want to read real implementations, start here: sklearn.tree.DecisionTreeClassifier</code> for tree splits, impurity, and pruning logic.</li> sklearn.ensemble.RandomForestClassifier</code> for bagging and feature subsampling.</li> xgboost.XGBClassifier</code> for second-order boosting with regularization.</li> </ul> The Big Picture</h2> We started with simple decision rules. That gave us decision trees. Decision trees are powerful but unstable, so we average them. That gives us bagging, and then Random Forests. Random Forests reduce variance but still leave bias on the table, so we build trees sequentially to correct errors. That gives us boosting, and XGBoost. Trees define nonlinear partitions, bagging stabilizes them, boosting makes them accurate. That is the whole story, and every formula above is one of those three moves written out. At some point I may extend the notebook with hyperparameter tuning and a multi-class example. If you want that sooner, leave a comment. Fast Fourier Transform Explained from the DFT, with a Sound-Wave Example 2026-04-22T00:00:00+00:00 The butterfly diagram is usually the first thing people show you about the FFT. I think it should be the last. The FFT is nothing more than a fast way to compute the Discrete Fourier Transform, so none of it makes sense until the DFT does. And the DFT itself is easiest to see if you start from a sampled sound wave. So this note goes: $$ \text{sound wave} \to \text{samples} \to \text{DFT} \to \text{FFT} \to \text{spectrum}. $$ We build up the picture of sound as a sum of sinusoids, get the DFT as its discrete version, and then let the structure of the DFT matrix hand us the Cooley-Tukey FFT almost for free. At the end we run the whole thing on a real sound wave, implemented from scratch in Python. Run it yourself. Every code block below is collected in a runnable Jupyter notebook: download fft-sound-wave.ipynb</code></a>. It only needs numpy</code> and matplotlib</code> (pip install numpy matplotlib</code>), and runs top to bottom. </blockquote> Start with a Sound Wave</h2> Think of a pure musical tone. When a tuning fork vibrates at $440$ Hz, it displaces air molecules in a nearly sinusoidal way. The pressure at a fixed point in space looks like $$ x(t) = A \cos(2\pi f t + \varphi), $$ with frequency $f = 440$ Hz, some amplitude $A$, and some phase $\varphi$. A real sound is almost never a single sinusoid. A piano note at A4 contains the fundamental at $440$ Hz plus overtones at roughly $880$ Hz, $1320$ Hz, and so on, each with its own amplitude. More generally, any reasonable sound can be written as a sum of sinusoids: $$ x(t) = \sum_{k} A_k \cos(2\pi f_k t + \varphi_k). $$ This is the Fourier picture. The time-domain view tells you what the waveform looks like at each instant. The frequency-domain view tells you which sinusoids, at which amplitudes and phases, combine to produce that waveform. The two views carry the same information but answer different questions. On a computer, we never see $x(t)$ directly. We see samples. An analogue-to-digital converter records the pressure at a fixed sample rate $f_s$ (for example $44{,}100$ samples per second for CD audio), producing a finite sequence $$ x_n = x(n / f_s), \qquad n = 0, 1, \ldots, N-1. $$ The question we really want to answer is: given these $N$ samples, which frequencies are present and how strong are they? That question has a precise answer called the Discrete Fourier Transform. From Fourier Series to the DFT</h2> The continuous Fourier series says that a periodic signal can be written as a sum of complex exponentials: $$ x(t) = \sum_{k=-\infty}^{\infty} c_k , e^{i 2\pi k t / T}, $$ where $T$ is the period. The coefficients $c_k$ tell us how much of each frequency $k/T$ is present. Complex exponentials show up here for a reason. Using Euler's formula, $$ e^{i\theta} = \cos\theta + i\sin\theta, $$ a single complex exponential carries both a cosine and a sine at the same frequency. That packages amplitude and phase into one object and lets the math stay linear and clean. Now discretize. Suppose we sample $x(t)$ at $N$ evenly spaced points over one period $T$, with spacing $\Delta t = T/N$. The samples are $$ x_n = x(n \Delta t), \qquad n = 0, 1, \ldots, N-1. $$ The natural discrete analogue of the Fourier series uses $N$ complex exponentials, $$ e^{i 2\pi k n / N}, \qquad k = 0, 1, \ldots, N-1, $$ as a basis for length-$N$ sequences. These discrete exponentials are orthogonal: $$ \sum_{n=0}^{N-1} e^{i 2\pi k n / N} , e^{-i 2\pi \ell n / N} = \begin{cases} N & \text{if } k = \ell, \ 0 & \text{otherwise.} \end{cases} $$ Orthogonality is what makes the transform invertible. It also tells us the right formula for the coefficients, by projecting $x_n$ onto each basis vector. This gives the Discrete Fourier Transform. For a sequence $x_0, x_1, \ldots, x_{N-1}$, define $$ X_k = \sum_{n=0}^{N-1} x_n , e^{-i 2\pi k n / N}, \qquad k = 0, 1, \ldots, N-1. $$ And the inverse: $$ x_n = \frac{1}{N} \sum_{k=0}^{N-1} X_k , e^{i 2\pi k n / N}. $$ Each $X_k$ is a complex number. Its magnitude $|X_k|$ tells us how strong the frequency component at bin $k$ is, and its phase $\arg X_k$ tells us where that sinusoid starts. The frequency associated with bin $k$ is $$ f_k = \frac{k}{N} f_s, $$ for $k = 0, 1, \ldots, N/2$, after which the bins fold by symmetry for real inputs. The highest frequency we can resolve is $f_s / 2$, the Nyquist frequency. Anything above that aliases down into lower bins, which is why sample rate matters. What the Transform Actually Does: The Winding View</h2> The formula for $X_k$ is easy to write down but easy to stare at without seeing anything. Before we optimize how to compute it, here is a different perspective on what it computes, one that makes the whole transform intuitive. Look again at a single term. The factor $$ e^{-i 2\pi k n / N} $$ is a unit vector in the complex plane that rotates clockwise as $n$ advances. Multiplying the signal $x_n$ by it winds the signal around the origin at a winding rate set by $k$: small $k$ winds slowly, large $k$ winds fast. The sum over $n$ then just adds up all those wound points, and dividing by $N$ gives their center of mass (their average position). So $X_k / N$ is literally: take the signal, wrap it around a circle at rate $k$, and find the center of mass of the resulting curve. Now the key effect. When the winding rate matches a frequency that is actually present in the signal, the signal's peaks land on the same side of the circle on every turn. The wound points pile up on one side, and their center of mass sits far from the origin: a large $|X_k|$. When the winding rate does not match, the peaks land at different angles each turn, spread evenly around the circle, and cancel. The center of mass collapses to near zero. To see it, here is a simple two-component signal (a $2$ Hz plus a $3$ Hz cosine, chosen at low frequencies purely so the winding is visible) wound at four different rates: Winding a 2 Hz + 3 Hz signal. At 2 Hz and 3 Hz the center of mass (red) pulls away from the origin; at 1 Hz and 2.5 Hz it cancels back to the center.</figcaption> </figure> Sweep the winding rate continuously and record how far the center of mass sits from the origin at each rate, and you trace out the Fourier transform magnitude directly: flat and near zero everywhere except sharp bumps at exactly the frequencies the signal contains. Sweeping the winding frequency reproduces the spectrum: peaks at 2 and 3 Hz, near-zero elsewhere. This curve *is* the Fourier transform.</figcaption> </figure> The same picture is clearer in three dimensions. Stack each wound copy of the signal along a third axis given by its winding frequency. Most slices are balanced loops centered on the axis; only at the true frequencies does the center of mass bulge outward. The same winding viewed in 3D. Each slice is the signal wound at one frequency; the red centers of mass bulge out only at 2 and 3 Hz.</figcaption> </figure> That is the one-sentence summary of the entire transform: the DFT measures how well the signal lines up with a corkscrew of each frequency. The complex exponential $e^{-i 2\pi k n / N}$ is that corkscrew, and the sum is the alignment score. Everything that follows (the matrix form, the even/odd split, the butterflies) is just machinery for computing all $N$ of these alignment scores quickly. The DFT as a Matrix</h2> Define the primitive root of unity $$ \omega_N = e^{-i 2\pi / N}. $$ Then $$ X_k = \sum_{n=0}^{N-1} x_n , \omega_N^{kn}. $$ Let $F_N$ be the $N \times N$ matrix with entries $(F_N)_{k,n} = \omega_N^{kn}$. Then the DFT is just a matrix-vector product: $$ X = F_N x. $$ A matrix-vector product with a dense $N \times N$ matrix takes $O(N^2)$ multiplications and additions. For a one-second CD-quality audio frame with $N = 44{,}100$ samples that is about $2 \times 10^9$ operations per frame. Doable, but clearly wasteful if the matrix has structure. It does. The matrix $F_N$ is extremely structured, because $\omega_N$ is a root of unity. The FFT is the algorithm that exploits that structure. The FFT Idea: Split Into Even and Odd</h2> The Cooley-Tukey FFT is a divide-and-conquer algorithm. Assume $N$ is a power of two (we can always zero-pad to get there). Split the input sequence by index parity: $$ \text{even part: } ; x_0, x_2, x_4, \ldots, x_{N-2}, $$ $$ \text{odd part: } ; x_1, x_3, x_5, \ldots, x_{N-1}. $$ Now rewrite $X_k$ by separating even and odd indices: $$ X_k ;=; \sum_{m=0}^{N/2 - 1} x_{2m} , e^{-i 2\pi k (2m) / N} ;+; \sum_{m=0}^{N/2 - 1} x_{2m+1} , e^{-i 2\pi k (2m+1) / N}. $$ Pull out the extra factor in the odd sum: $$ X_k ;=; \sum_{m=0}^{N/2 - 1} x_{2m} , e^{-i 2\pi k m / (N/2)} ;+; e^{-i 2\pi k / N} \sum_{m=0}^{N/2 - 1} x_{2m+1} , e^{-i 2\pi k m / (N/2)}. $$ Now look carefully at each sum. Each one is itself a DFT, of length $N/2$. Let $$ E_k = \sum_{m=0}^{N/2 - 1} x_{2m} , e^{-i 2\pi k m / (N/2)}, \qquad O_k = \sum_{m=0}^{N/2 - 1} x_{2m+1} , e^{-i 2\pi k m / (N/2)}. $$ Then $$ X_k = E_k + e^{-i 2\pi k / N} , O_k. $$ The quantity $e^{-i 2\pi k / N}$ is usually written $$ W_N^k = e^{-i 2\pi k / N} $$ and called a twiddle factor. It is the bookkeeping that corrects for the shift between even and odd indices. The Butterfly: Two Outputs for the Price of One</h2> We have computed $X_k$ in terms of $E_k$ and $O_k$ for $k = 0, 1, \ldots, N/2 - 1$. What about $k = N/2, N/2+1, \ldots, N-1$? Use the periodicity of $E_k$ and $O_k$. Both are length-$N/2$ DFTs, so $$ E_{k + N/2} = E_k, \qquad O_{k + N/2} = O_k. $$ The twiddle factor shifts by a sign: $$ W_N^{k + N/2} = e^{-i 2\pi (k + N/2) / N} = e^{-i 2\pi k / N} e^{-i\pi} = -W_N^k. $$ Combining these, $$ X_k = E_k + W_N^k , O_k, $$ $$ X_{k + N/2} = E_k - W_N^k , O_k. $$ That pair of equations is one butterfly. From two inputs $E_k$ and $O_k$ (and a precomputed twiddle), we produce two outputs $X_k$ and $X_{k+N/2}$ using one complex multiplication and two complex additions. We reuse work instead of redoing it. At this point the whole algorithm is already visible. Compute two $N/2$-point DFTs. Combine them with $N/2$ butterflies to produce the $N$-point DFT. Recurse on each half. Complexity: Why It Is $O(N \log N)$</h2> Let $T(N)$ be the number of operations to compute an $N$-point DFT using this recursion. We do: Two DFTs of size $N/2$, each costing $T(N/2)$.</li> $N/2$ butterflies to combine them, each a constant amount of work. So $O(N)$ total for the combine step.</li> </ul> That gives the recurrence $$ T(N) = 2 T(N/2) + cN, $$ with $T(1) = O(1)$, for some constant $c$. Unroll the recurrence: $$ T(N) ;=; 2T(N/2) + cN ;=; 4 T(N/4) + 2cN ;=; 8 T(N/8) + 3cN ;=; \ldots $$ After $\log_2 N$ levels of unrolling we reach $T(1)$, and each level contributes $cN$ work: $$ T(N) = N \cdot T(1) + cN \log_2 N = O(N \log N). $$ Compare that to the naive DFT. For $N = 2^{20} \approx 10^6$: $$ N^2 = 10^{12}, \qquad N \log_2 N \approx 2 \times 10^7. $$ Five orders of magnitude. That is the whole reason real-time audio processing, modern communications, and large-scale scientific computing are tractable. A Worked Example: N = 4</h2> To see the recursion concretely, take $N = 4$ with input $x = (x_0, x_1, x_2, x_3)$. The even part is $(x_0, x_2)$, and the odd part is $(x_1, x_3)$. Both are length-2 DFTs. For a length-2 DFT with input $(a, b)$: $$ Y_0 = a + b, \qquad Y_1 = a - b, $$ which is itself one butterfly with twiddle $W_2^0 = 1$. So the two half-DFTs give $$ E_0 = x_0 + x_2, \quad E_1 = x_0 - x_2, $$ $$ O_0 = x_1 + x_3, \quad O_1 = x_1 - x_3. $$ The twiddle factors at size $N=4$ are $W_4^0 = 1$ and $W_4^1 = e^{-i\pi/2} = -i$. Combining: $$ X_0 = E_0 + W_4^0 , O_0 = (x_0 + x_2) + (x_1 + x_3), $$ $$ X_1 = E_1 + W_4^1 , O_1 = (x_0 - x_2) - i(x_1 - x_3), $$ $$ X_2 = E_0 - W_4^0 , O_0 = (x_0 + x_2) - (x_1 + x_3), $$ $$ X_3 = E_1 - W_4^1 , O_1 = (x_0 - x_2) + i(x_1 - x_3). $$ Four outputs, built from two pairs of butterflies. You can check by hand that these match $X_k = \sum_n x_n \omega_4^{kn}$ directly. This is the full FFT, in miniature. Hand Calculation with Real Numbers</h2> The symbolic form is useful, but it is more convincing to watch the recursion run on actual numbers. Take the concrete input $$ x = (x_0, x_1, x_2, x_3) = (1, 2, 3, 4). $$ Following the principle from the previous section step by step. Step 1. Split by parity. $$ \text{even: } (x_0, x_2) = (1, 3), \qquad \text{odd: } (x_1, x_3) = (2, 4). $$ Step 2. Length-2 DFT of each half. Using $Y_0 = a + b$, $Y_1 = a - b$: $$ E_0 = 1 + 3 = 4, \qquad E_1 = 1 - 3 = -2, $$ $$ O_0 = 2 + 4 = 6, \qquad O_1 = 2 - 4 = -2. $$ Step 3. Twiddle factors at $N = 4$. $$ W_4^0 = e^{0} = 1, \qquad W_4^1 = e^{-i\pi/2} = -i. $$ Step 4. Apply the butterfly equations. Using $X_k = E_k + W_4^k O_k$ and $X_{k+2} = E_k - W_4^k O_k$: $$ X_0 = E_0 + W_4^0 , O_0 = 4 + (1)(6) = 10, $$ $$ X_1 = E_1 + W_4^1 , O_1 = -2 + (-i)(-2) = -2 + 2i, $$ $$ X_2 = E_0 - W_4^0 , O_0 = 4 - 6 = -2, $$ $$ X_3 = E_1 - W_4^1 , O_1 = -2 - (-i)(-2) = -2 - 2i. $$ So by hand, $$ X = (10,; -2 + 2i,; -2,; -2 - 2i). $$ Step 5. Cross-check with code. Running the same input through numpy</code> and our from-scratch implementations should produce exactly the same vector: import numpy as np x = np.array([1, 2, 3, 4], dtype=complex) print("numpy :", np.fft.fft(x)) print("naive :", naive_dft(x)) print("fft_rec:", fft_recursive(x))</code></pre> Expected output (up to floating-point error): numpy : [10.+0.j -2.+2.j -2.+0.j -2.-2.j] naive : [10.+0.j -2.+2.j -2.+0.j -2.-2.j] fft_rec: [10.+0.j -2.+2.j -2.+0.j -2.-2.j]</code></pre> Three different routes, one answer. The naive DFT sums all $N^2 = 16$ terms directly. The recursive FFT computes two length-2 DFTs and combines them with two butterflies, using far fewer operations. Both recover the same spectrum because they are computing the same mathematical object, just through different arithmetic paths. This is the moment where the principle becomes tangible. Even/odd split, two half-DFTs, twiddle factor, butterfly. Four equations, one vector of outputs. Scale the same pattern recursively to $N = 8, 16, \ldots, 2^{20}$ and you have the full FFT. From Scratch in Python: Naive DFT and Recursive FFT</h2> The fastest way to internalize this is to implement both and watch them agree. Start with the naive DFT, a direct translation of the definition: import numpy as np def naive_dft(x): x = np.asarray(x, dtype=complex) N = x.shape[0] n = np.arange(N) k = n.reshape((N, 1)) M = np.exp(-2j * np.pi * k * n / N) return M @ x</code></pre> Now the recursive Cooley-Tukey FFT. It assumes $N$ is a power of two. When $N$ drops to a small base case we hand off to the naive DFT, which is fine for tiny sizes: def fft_recursive(x): x = np.asarray(x, dtype=complex) N = x.shape[0] if N & (N - 1): raise ValueError("Length of input must be a power of 2.") if N <= 2: return naive_dft(x) X_even = fft_recursive(x[0::2]) X_odd = fft_recursive(x[1::2]) k = np.arange(N // 2) twiddle = np.exp(-2j * np.pi * k / N) return np.concatenate([X_even + twiddle * X_odd, X_even - twiddle * X_odd])</code></pre> Check agreement with numpy.fft.fft</code> on random input: rng = np.random.default_rng(0) x = rng.standard_normal(1024) X_naive = naive_dft(x) X_fft = fft_recursive(x) X_np = np.fft.fft(x) print(np.allclose(X_naive, X_np)) print(np.allclose(X_fft, X_np))</code></pre> Both should print True</code> up to floating-point error. The three implementations compute the same mathematical object. The FFT just computes it faster. A Real Sound-Wave Example</h2> Now the payoff. We construct a synthetic sound wave made of three tones, sample it, and recover those tones from the samples using the FFT. import numpy as np import matplotlib.pyplot as plt sample_rate = 8000 duration = 1.0 N = int(sample_rate * duration) t = np.arange(N) / sample_rate x = (1.0 * np.sin(2 * np.pi * 220 * t) + 0.6 * np.sin(2 * np.pi * 440 * t) + 0.3 * np.sin(2 * np.pi * 880 * t)) x += 0.05 * np.random.default_rng(0).standard_normal(N)</code></pre> The signal x</code> contains a strong $220$ Hz tone, a medium $440$ Hz tone, a weaker $880$ Hz tone, and a small amount of noise. In the time domain the waveform looks periodic but visually messy. That is expected. Time domain is not the right view for this question. A short window of the synthetic three-tone signal in the time domain.</figcaption> </figure> Now apply the FFT. Because N = 8000</code> is not a power of two, we either zero-pad up to the next power of two or use a mixed-radix FFT. For this note we will zero-pad, which is also a common practical trick to improve bin resolution: def next_pow2(n): return 1 << (n - 1).bit_length() N_pad = next_pow2(N) x_pad = np.zeros(N_pad) x_pad[:N] = x X = fft_recursive(x_pad) magnitude = np.abs(X) / N freqs = np.arange(N_pad) * sample_rate / N_pad half = N_pad // 2 plt.plot(freqs[:half], magnitude[:half]) plt.xlabel("Frequency (Hz)") plt.ylabel("Magnitude") plt.title("Magnitude spectrum of the synthetic sound wave") plt.grid(True) plt.show()</code></pre> The spectrum shows three clean peaks, at $220$, $440$, and $880$ Hz, with heights roughly proportional to the amplitudes we put in. The noise spreads thinly across all other bins and stays small. The DFT magnitude spectrum. The three input sinusoids appear as isolated peaks at 220, 440, and 880 Hz.</figcaption> </figure> That is the central picture. The time-domain view showed an oscillating mess. The frequency-domain view recovers exactly the three tones we synthesized, because that is what the DFT does. The FFT just made the computation fast enough to be practical on a full second of audio. Naive DFT vs FFT: Timing</h2> To make the complexity story concrete, it helps to time both implementations on increasing input sizes. The naive DFT should grow quadratically, and the recursive FFT should grow close to linearly in $N \log N$: import time sizes = [2**k for k in range(6, 13)] naive_times = [] fft_times = [] for N in sizes: x = np.random.default_rng(0).standard_normal(N) t0 = time.perf_counter() naive_dft(x) naive_times.append(time.perf_counter() - t0) t0 = time.perf_counter() fft_recursive(x) fft_times.append(time.perf_counter() - t0) plt.loglog(sizes, naive_times, marker="o", label="naive DFT") plt.loglog(sizes, fft_times, marker="o", label="recursive FFT") plt.xlabel("N") plt.ylabel("Time (s)") plt.title("Runtime: naive DFT vs recursive FFT") plt.legend() plt.grid(True, which="both") plt.show()</code></pre> On a log-log plot, the naive DFT has slope close to $2$ and the recursive FFT has slope close to $1$. That is the $O(N^2)$ versus $O(N \log N)$ gap, visualized directly. Empirical runtime comparison. The naive DFT grows quadratically while the recursive FFT grows close to linearly in N log N.</figcaption> </figure> Sampling, Nyquist, and Aliasing</h2> A few practical points are worth making explicit, because they trip people up in real audio code. The highest frequency the DFT can represent is the Nyquist frequency $f_s / 2$. At sample rate $8000$ Hz, that is $4000$ Hz. Any signal component above that frequency does not simply vanish. It aliases down into a lower bin and corrupts the spectrum. That is why audio is low-pass filtered before sampling. For real-valued inputs, the spectrum is conjugate-symmetric: $X_{N-k} = \overline{X_k}$. So the second half of the DFT output carries no new information. In practice we only plot bins $0$ through $N/2$. The frequency resolution of the DFT is $f_s / N$. Longer windows give finer frequency resolution but worse time resolution. This trade-off is exactly why short-time Fourier transforms and wavelets exist. Windowing and Spectral Leakage</h2> If the signal frequency does not land exactly on a DFT bin, its energy spreads into neighboring bins. This is called spectral leakage, and it is a property of the DFT, not a bug. Leakage is reduced by multiplying the signal by a smooth window function before the transform: window = np.hanning(N) x_windowed = x * window</code></pre> A Hann window tapers the signal smoothly to zero at both ends, which suppresses the discontinuity that causes leakage. The trade-off is a slight widening of each spectral peak. For most audio applications this is a good deal. Real Input: A Slightly Faster Variant</h2> When the input is real, numpy.fft.rfft</code> returns only the non-redundant half of the spectrum and is about twice as fast as the full complex FFT. In from-scratch code the same idea can be implemented by exploiting the conjugate symmetry directly. For production use, numpy.fft.rfft</code> or any mature FFT library (FFTW, PocketFFT, cuFFT) is almost always the right choice. The recursive implementation above is for understanding, not for speed. The Big Picture</h2> At this point the whole thing should feel like one continuous argument, not three unrelated algorithms. We started with a sound wave in the time domain and asked which frequencies it contains. Sampling gave us a finite sequence $x_n$. Projecting that sequence onto a basis of complex exponentials gave the DFT: $$ X_k = \sum_{n=0}^{N-1} x_n , e^{-i 2\pi k n / N}. $$ The DFT is a linear map, so it can be written as a matrix product $X = F_N x$. Computed naively, that is $O(N^2)$. We then exploited the structure of $F_N$. The roots of unity satisfy a recursion, so the $N$-point DFT splits into two $N/2$-point DFTs combined by $N/2$ butterflies. That gives the recurrence $T(N) = 2T(N/2) + O(N)$ and the complexity $O(N \log N)$. Finally, we tested the whole pipeline on a synthetic sound wave and recovered the three input tones as three clean peaks in the magnitude spectrum. So in the end, the DFT is the definition and the FFT is the algorithm. The DFT says what we want. The FFT says how to get it quickly. The exponential keeps showing up for the same reason it does in logistic and softmax regression: it is the right basis when the underlying object has multiplicative structure. Roots of unity are multiplicative. Splitting them in half, again and again, is exactly what turns $N^2$ into $N \log N$. If I keep going with this, the next stops are the short-time Fourier transform and spectrograms, and after that the jump from Fourier bases to wavelets. Corrections and better examples are welcome in the comments. Softmax Regression Explained from Logistic Regression 2026-04-19T00:00:00+00:00 Softmax regression is what logistic regression grows into when there are more than two classes, and logistic regression itself only really makes sense once you understand odds. So instead of starting at the softmax formula, this note starts all the way back at odds and walks forward: $$ \text{probability} \to \text{odds} \to \text{log-odds} \to \text{logistic regression} \to \text{softmax regression}. $$ By the end, the exponential should not feel like a trick anymore. It is just the machine that turns unrestricted linear scores into positive ratios. If you want the full runnable walkthrough, the companion notebook lives on Kaggle: https://www.kaggle.com/code/lyhatt/softmax-from-logistic-regression</a> That notebook was assisted by AI tools. This note is the written version. Start with Probability, Then Ask for More Structure</h2> Suppose we have a binary outcome. Let $$ p = P(y=1 \mid x) $$ be the probability that the positive class occurs given the input $x$. Probability is easy to interpret, but it is not yet the best thing to model. A slightly more structured object is the odds: $$ \text{odds} = \frac{p}{1-p}. $$ The odds tell us how much more likely the positive event is than the negative one. Some quick examples help. If $p = 0.5$, then the odds are $1$. If $p = 0.8$, then the odds are $\frac{0.8}{0.2} = 4$. If $p = 0.2$, then the odds are $\frac{0.2}{0.8} = 0.25$. So odds greater than $1$ favor the positive class, odds less than $1$ favor the negative class, and odds equal to $1$ mean the two classes are balanced. This is already useful, but there is a problem. Odds are always positive, so they live on $(0,\infty)$. A plain linear model $$ w^T x + b $$ lives on the whole real line, from $-\infty$ to $\infty$. Those two spaces do not match well. If we want a linear model, odds are the wrong coordinates. The Core Move: Take Logarithms</h2> To fix this, we take the logarithm of the odds: $$ \log \frac{p}{1-p}. $$ This quantity is the log-odds, also called the logit. Why is this useful? Because odds are constrained to be positive, but log-odds can be any real number. A logit of $0$ corresponds to $p=0.5$, positive logits correspond to probabilities above $0.5$, and negative logits correspond to probabilities below $0.5$. This is the key move. The logarithm converts a positive ratio into a quantity that can now range over the whole real line. That means a linear model suddenly fits naturally: $$ \log \frac{p}{1-p} = w^T x + b. $$ This is logistic regression in its cleanest form. The model says: not that the probability is linear, but that the log-odds are linear. Now Solve for the Probability</h2> Now let us solve for $p$ carefully. Start with $$ \log \frac{p}{1-p} = w^T x + b. $$ Exponentiate both sides: $$ \frac{p}{1-p} = e^{w^T x + b}. $$ This is where the exponential enters the story, and it is doing real mathematical work. It appears because it is the inverse of the logarithm, because it guarantees the odds stay positive, and because it converts an unrestricted real-valued score into a valid positive ratio. Now multiply both sides by $(1-p)$: $$ p = (1-p)e^{w^T x + b}. $$ Expand the right-hand side: $$ p = e^{w^T x + b} - pe^{w^T x + b}. $$ Collect the $p$ terms: $$ p + pe^{w^T x + b} = e^{w^T x + b}. $$ Factor out $p$: $$ p\left(1 + e^{w^T x + b}\right) = e^{w^T x + b}. $$ So $$ p = \frac{e^{w^T x + b}}{1 + e^{w^T x + b}}. $$ Divide numerator and denominator by $e^{w^T x + b}$: $$ p = \frac{1}{1 + e^{-(w^T x + b)}}. $$ This is the sigmoid form of logistic regression: $$ p = \sigma(w^T x + b). $$ This is the full derivation. The sigmoid is not a decorative choice. It is what falls out when a linear model is placed on the log-odds. What the Sigmoid Is Really Doing</h2> The sigmoid takes a real-valued score and maps it to a number between $0$ and $1$: $$ \sigma(z) = \frac{1}{1+e^{-z}}. $$ Here $z = w^T x + b$ is often called the logit. The interpretation is immediate once you remember that $z$ is a logit. If $z \gg 0$, then $\sigma(z)$ is close to $1$. If $z = 0$, then $\sigma(z)=0.5$. If $z \ll 0$, then $\sigma(z)$ is close to $0$. So the sigmoid is the map from linear evidence to probability. That is why logistic regression has the familiar S-curve. The sigmoid curve appears after solving the log-odds equation for the probability.</figcaption> </figure> The Model Needs a Learning Signal</h2> Once the model outputs probabilities, the next question is: what objective should it optimize? Logistic regression uses the Bernoulli likelihood. For one example with label $y \in {0,1}$ and predicted probability $\hat{y}$, $$ P(y \mid x) = \hat{y}^y(1-\hat{y})^{1-y}. $$ Taking logs gives the log-likelihood: $$ \log P(y \mid x) = y \log \hat{y} + (1-y)\log(1-\hat{y}). $$ For $m$ observations, the negative average log-likelihood becomes the binary cross-entropy: $$ L = -\frac{1}{m} \sum_{i=1}^m \left[y^{(i)} \log \hat{y}^{(i)} + (1-y^{(i)})\log(1-\hat{y}^{(i)})\right]. $$ This loss is worth staring at for a moment. It tells you exactly what the model fears: confident mistakes. Binary cross-entropy for the two possible true labels.</figcaption> </figure> That is the right behavior for a probabilistic model. If it is unsure, the penalty is moderate. If it is confidently wrong, the penalty should explode. The Optimization Is Still Just Gradient Descent</h2> Once we define the loss, the training rule is still gradient descent. The gradients are $$ \nabla_w L = \frac{1}{m}X^T(\hat{y}-y), \qquad \nabla_b L = \frac{1}{m}\sum_{i=1}^{m}(\hat{y}^{(i)} - y^{(i)}). $$ So the update step is $$ w \leftarrow w - \alpha \nabla_w L, \qquad b \leftarrow b - \alpha \nabla_b L. $$ This is worth emphasizing. Logistic regression sounds like a special model class, but algorithmically it is still simple: define probabilities, define a loss, differentiate, and descend. A Real Binary Example</h2> To make the binary story concrete, we use the Breast Cancer Wisconsin dataset. It has 569 samples, 30 numerical features, and two target classes: 0</code> for malignant and 1</code> for benign. Before fitting anything, it helps to get a feel for the data itself. Class balance and a two-feature raw view of the Breast Cancer dataset.</figcaption> </figure> This is just the intuition view. The real training run uses all standardized features, not only the two shown here. Below is a compact from-scratch implementation of logistic regression with gradient descent: import numpy as np def sigmoid(z): return 1.0 / (1.0 + np.exp(-z)) def binary_cross_entropy(y_true, y_prob): eps = 1e-12 y_prob = np.clip(y_prob, eps, 1 - eps) return -np.mean(y_true * np.log(y_prob) + (1 - y_true) * np.log(1 - y_prob)) def train_binary_logistic(X, y, learning_rate=0.1, epochs=1500): n_samples, n_features = X.shape weights = np.zeros(n_features) bias = 0.0 history = [] for _ in range(epochs): logits = X @ weights + bias probs = sigmoid(logits) error = probs - y grad_w = X.T @ error / n_samples grad_b = np.mean(error) weights -= learning_rate * grad_w bias -= learning_rate * grad_b history.append(binary_cross_entropy(y, probs)) return weights, bias, np.array(history)</code></pre> The training loss falls in the way we expect: Training loss for from-scratch logistic regression.</figcaption> </figure> If we restrict ourselves to two features, we can also draw the probability surface directly: A two-feature probability surface for logistic regression.</figcaption> </figure> That is enough for the binary story. The point is not to pile up diagnostics. The point is to see that logistic regression turns a linear score into a probability, and then learns that mapping through gradient descent. The Jump to Many Classes</h2> Now comes the real question: what changes when there are not two classes, but many? In binary logistic regression, we can think of the model as comparing two outcomes. The odds are $$ \frac{p}{1-p}. $$ That ratio is already a comparison between classes. Softmax keeps the same spirit, but scales it from two classes to $K$ classes. One useful way to think about this is to stop thinking in terms of labels first and think in terms of score comparisons. Imagine an image classifier. A certain image might look a bit like a car, somewhat like a truck, and definitely not like a frog. In that situation, the model does not begin with probabilities. It begins with raw scores. Maybe the car score is high, the truck score is also fairly high, and the frog score is very low. That is already a meaningful picture. "Car versus truck" is a hard distinction because those classes can share visual structure. "Car versus frog" is easier because the scores should separate much more aggressively. Softmax takes exactly this kind of score vector and says: fine, now turn those relative preferences into a probability distribution. Suppose each class $k$ gets its own score $$ z_k. $$ If we want a multiclass probability model, the outputs should satisfy three requirements. Each probability must be positive, all probabilities must sum to $1$, and a larger score should correspond to a larger probability. The exponential is the natural choice here because $e^{z_k}$ is always positive, because if $z_a > z_b$ then $e^{z_a} > e^{z_b}$, and because exponentials turn unrestricted scores into positive quantities that can be normalized. So we first define unnormalized class weights: $$ \tilde{p}_k = e^{z_k}. $$ These are not probabilities yet, because they do not sum to $1$. So we normalize them: $$ p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}. $$ That normalization step is softmax. Why Exponentials Keep Showing Up</h2> At this point the exponential should no longer feel mysterious. For logistic regression, we start from log-odds, exponentiation recovers the odds, and the final algebra gives the sigmoid probability. For softmax regression, we start from a score for each class, exponentiation turns each score into a positive class weight, and normalization turns those positive weights into a probability distribution. It is the same structural tool in both models: take unrestricted scores, convert them into positive quantities, and only then interpret them probabilistically. There is also an intuitive reason practitioners like this view. A difference in score should matter more when it is decisive and less when two classes are nearly tied. Exponentials amplify those differences in a smooth way. If "car" scores only slightly above "truck", the probabilities stay competitive. If "car" scores far above "frog", the probability mass shifts much more aggressively. Softmax Actually Contains Logistic Regression</h2> Softmax is the multiclass extension of logistic regression, and we can show that mathematically. Take the two-class case with scores $z_0$ and $z_1$. Then softmax gives $$ p_1 = \frac{e^{z_1}}{e^{z_0} + e^{z_1}}. $$ Factor out $e^{z_1}$ in the denominator: $$ p_1 = \frac{e^{z_1}}{e^{z_1}\left(e^{z_0-z_1}+1\right)} = \frac{1}{1 + e^{z_0-z_1}}. $$ Equivalently, $$ p_1 = \frac{1}{1 + e^{-(z_1-z_0)}}. $$ This is exactly a sigmoid in the score difference $z_1-z_0$. That is the bridge. Logistic regression is the binary special case, and softmax regression is the multiclass generalization. The Multiclass Loss Is the Same Idea Again</h2> In the multiclass setting, the true label is represented as a one-hot vector. If $Y$ is the one-hot target matrix and $P$ is the predicted softmax probability matrix, then the multiclass cross-entropy is $$ L = -\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K} y_k^{(i)} \log p_k^{(i)}. $$ This is the same negative log-likelihood idea again, just written for many classes instead of two. The gradient keeps a remarkably clean form: $$ \nabla_W L = \frac{1}{m}X^T(P-Y), \qquad \nabla_b L = \frac{1}{m}\sum_{i=1}^{m}(P^{(i)} - Y^{(i)}). $$ So the optimization rule is still gradient descent: $$ W \leftarrow W - \alpha \nabla_W L, \qquad b \leftarrow b - \alpha \nabla_b L. $$ So the whole picture survives the transition almost unchanged. The binary weight vector becomes a class-wise weight matrix, and everything else follows. A Real Multiclass Example</h2> For the multiclass example, we use the Digits dataset. This is much more appropriate for softmax because it contains ten classes instead of two. Each digit image is an $8 \times 8$ grayscale grid, flattened into a 64-dimensional vector. Before fitting the model, it is worth seeing both raw samples and class averages: Raw digits samples and average digit images by class.</figcaption> </figure> Here is a compact from-scratch softmax regression implementation: import numpy as np def softmax(logits): shifted = logits - np.max(logits, axis=1, keepdims=True) exp_logits = np.exp(shifted) return exp_logits / np.sum(exp_logits, axis=1, keepdims=True) def multiclass_cross_entropy(y_one_hot, probs): eps = 1e-12 probs = np.clip(probs, eps, 1 - eps) return -np.mean(np.sum(y_one_hot * np.log(probs), axis=1)) def train_softmax_regression(X, y, learning_rate=0.1, epochs=3000): n_samples, n_features = X.shape n_classes = len(np.unique(y)) weights = np.zeros((n_features, n_classes)) bias = np.zeros(n_classes) y_one_hot = np.eye(n_classes)[y] history = [] for _ in range(epochs): logits = X @ weights + bias probs = softmax(logits) error = probs - y_one_hot grad_w = X.T @ error / n_samples grad_b = np.mean(error, axis=0) weights -= learning_rate * grad_w bias -= learning_rate * grad_b history.append(multiclass_cross_entropy(y_one_hot, probs)) return weights, bias, np.array(history)</code></pre>How to Visualize Softmax Honestly</h2> Sigmoid has a universal curve. Softmax does not. So the honest softmax visualization is not a fake one-dimensional curve. It is a sample-level transformation: start from a real digit image, compute one raw logit per class, and then exponentiate and normalize into probabilities. That is exactly what the next figure shows: A real digit mapped into logits and then into softmax probabilities.</figcaption> </figure> That is the mental model to keep: softmax takes a vector of scores and turns it into a probability distribution. In other words, softmax is less about drawing one magical curve and more about resolving competitions among classes. Some competitions are close, like car versus truck. Some are not, like car versus frog. The probability vector tells us how that competition ended for one specific input. The training loss on Digits decreases as expected: Training loss for from-scratch softmax regression on Digits.</figcaption> </figure> That is enough for the multiclass side as well. Once the reader sees logits become probabilities on a real sample, the central idea has already landed. The remaining plots are useful in a notebook, but they are not necessary in the main essay. A Few Practical Tradeoffs</h2> This is also a good place to be honest about what logistic regression and softmax regression can and cannot do. They are simple, fast, interpretable, and they produce probabilities directly, which is often exactly what we want. They also work surprisingly well when the classes are close to linearly separable in feature space. But the tradeoff is clear: if the real boundary is highly nonlinear, these models will struggle, the quality of the features matters a great deal, and multiclass confusion often comes from genuinely similar classes rather than from a bug in the formula. This is also where people usually start reaching for nonlinear models. In deep learning, nonlinear functions become popular precisely because they can build more complex decision boundaries than a single linear logit layer can express. They are often used as activation functions inside deeper networks, and that added nonlinearity is what lets the model represent more complicated structure. But that is really another topic. Here the point is to understand the clean linear baseline first. That is why these models are such good teaching models and such good baselines. When they succeed, they do so for understandable reasons. When they fail, they usually fail in ways that teach us something about the data. The Big Picture</h2> At this point the whole thing should feel like one continuous argument, not three unrelated formulas. We started with probability, then moved to odds: $$ \frac{p}{1-p}. $$ We took logarithms to obtain log-odds: $$ \log \frac{p}{1-p}. $$ We modeled the log-odds linearly and solved for the probability, which produced the sigmoid and logistic regression. Then we asked what happens when there are many classes instead of two. The same need appears again: we must turn unrestricted scores into valid positive ratios, and then normalize them into probabilities. That requirement leads directly to the softmax formula. So logistic regression is a model for binary log-odds, and softmax regression is the same idea with more classes. The exponential appears because it converts linear scores into positive ratios, and gradient descent is the engine underneath both. Learn them together; separately they both look more arbitrary than they are. Natural continuations from here: cross-entropy from maximum likelihood, and the step from softmax regression to a neural network, which is just softmax with learned features. Corrections welcome in the comments. Statistical Testing in Regression 2026-02-20T00:00:00+00:00 This note covers hypothesis testing for regression coefficients and model terms, with emphasis on what p-values can and cannot tell you. The practical problem is making defensible decisions under uncertainty without overclaiming. I focus on test setup, assumptions, and interpretation boundaries. In the previous post</a>, we constructed confidence intervals for regression coefficients. Confidence intervals and hypothesis tests are two sides of the same coin. In this post, we formalize the hypothesis testing framework for regression coefficients. Hypothesis Testing for Individual Coefficients</h2> The most common test in regression asks whether a particular predictor has a significant linear effect on the response. For coefficient $\beta_j$, the hypotheses are: $$H_0: \beta_j = 0 \quad \text{(the predictor has no effect)},$$ $$H_1: \beta_j \neq 0 \quad \text{(the predictor has an effect)}.$$ This is a two-sided test. One-sided alternatives ($H_1: \beta_j > 0$ or $H_1: \beta_j < 0$) are also possible when the direction of the effect is of interest. The t-Test Statistic</h2> Under $H_0: \beta_j = 0$, the test statistic is: $$t = \frac{\hat{\beta}_j - 0}{SE(\hat{\beta}_j)} = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)}.$$ Under the null hypothesis and the normality assumption, this statistic follows a t-distribution: $$t \sim t_{n-p},$$ where $n - p$ is the residual degrees of freedom. For Simple Linear Regression</h3> Testing $H_0: \beta_1 = 0$ (the slope is zero): $$t = \frac{\hat{\beta}_1}{SE(\hat{\beta}_1)} = \frac{\hat{\beta}_1}{\sqrt{MSE/S_{xx}}} = \frac{S_{xy}/S_{xx}}{\sqrt{MSE/S_{xx}}} = \frac{S_{xy}}{\sqrt{MSE \cdot S_{xx}}}.$$ This statistic follows $t_{n-2}$ under $H_0$. Testing $H_0: \beta_0 = 0$ (the intercept is zero): $$t = \frac{\hat{\beta}_0}{SE(\hat{\beta}_0)} = \frac{\hat{\beta}_0}{\sqrt{MSE(1/n + \bar{x}^2/S_{xx})}}.$$ For Multiple Regression</h3> Testing $H_0: \beta_i = 0$ for any coefficient $\beta_i$: $$t = \frac{\hat{\beta}_i}{SE(\hat{\beta}_i)} = \frac{\hat{\beta}_i}{\sqrt{MSE \cdot C_{ii}}},$$ where $C_{ii}$ is the appropriate diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$. Decision Rule</h2> For a significance level $\alpha$ (commonly $\alpha = 0.05$): Reject $H_0$ if $|t| > t_{\alpha/2, n-p}$. Fail to reject $H_0$ if $|t| \leq t_{\alpha/2, n-p}$. Rejecting $H_0$ means we have sufficient evidence that $\beta_j \neq 0$, suggesting that the predictor $x_j$ has a statistically significant effect on $y$. P-Values</h2> The p-value provides a continuous measure of the evidence against $H_0$. For a two-sided test: $$p\text{-value} = 2 \cdot P(T > |t_{\text{obs}}|),$$ where $T \sim t_{n-p}$ and $t_{\text{obs}}$ is the observed test statistic. Interpretation: The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming $H_0$ is true.</li> A small p-value (typically $< 0.05$) provides evidence against $H_0$.</li> A large p-value does not prove $H_0$ is true; it simply means the data do not provide strong evidence against it.</li> </ul> The decision rule using p-values is equivalent to using critical values: reject $H_0$ if $p\text{-value} < \alpha$. Testing Against a Non-Zero Value</h2> More generally, we can test $H_0: \beta_j = \beta_{j,0}$ for any hypothesized value $\beta_{j,0}$: $$t = \frac{\hat{\beta}_j - \beta_{j,0}}{SE(\hat{\beta}_j)}.$$ The test for $\beta_j = 0$ is simply the special case where $\beta_{j,0} = 0$. Connection to Confidence Intervals</h2> There is a direct equivalence between hypothesis tests and confidence intervals: Reject $H_0: \beta_j = \beta_{j,0}$ at level $\alpha$ if and only if $\beta_{j,0}$ falls outside the $(1 - \alpha) \times 100%$ confidence interval for $\beta_j$.</li> Fail to reject $H_0$ if and only if $\beta_{j,0}$ falls inside the confidence interval.</li> </ul> This means that when we check whether a 95% confidence interval for $\beta_1$ contains zero, we are implicitly performing a t-test at the $\alpha = 0.05$ level. Testing the Correlation</h2> For simple linear regression, testing $H_0: \beta_1 = 0$ is equivalent to testing $H_0: \rho = 0$ (the population correlation is zero). This is because $\hat{\beta}_1 = r \cdot S_y/S_x$, and $\hat{\beta}_1 = 0$ if and only if $r = 0$. The test statistic can also be written in terms of $r$: $$t = r\sqrt{\frac{n - 2}{1 - r^2}},$$ which follows $t_{n-2}$ under $H_0: \rho = 0$. Practical vs. Statistical Significance</h2> A statistically significant result (small p-value) does not necessarily mean the result is practically important. With a large sample size, even a tiny effect can be statistically significant because the standard error shrinks with $n$. Conversely, a non-significant result does not mean the effect is zero; it may reflect insufficient sample size (low statistical power). When interpreting regression results, consider both: Statistical significance: Is the p-value small enough to reject $H_0$?</li> Practical significance: Is the estimated effect $\hat{\beta}_j$ large enough to matter in the context of the problem?</li> </ol> The confidence interval is particularly helpful here because it shows both the direction and the plausible range of the effect size. Multiple Testing Considerations</h2> When testing multiple coefficients in a regression model, each at level $\alpha$, the probability of at least one false rejection increases. If you test $k$ independent hypotheses at $\alpha = 0.05$, the family-wise error rate is approximately $1 - (1 - \alpha)^k$. Common corrections include: Bonferroni correction: Test each hypothesis at $\alpha/k$.</li> Holm-Bonferroni method: A sequential version that is less conservative.</li> False Discovery Rate (FDR) control: Controls the expected proportion of false rejections.</li> </ul> In regression, the overall model significance is typically assessed using the F-test (covered in the next post), which tests all predictors simultaneously. Summary</h2> In this post, we covered hypothesis testing for regression coefficients: The t-statistic $t = \hat{\beta}_j / SE(\hat{\beta}_j)$ follows $t_{n-p}$ under $H_0: \beta_j = 0$.</li> Reject $H_0$ when $|t| > t_{\alpha/2, n-p}$ or equivalently when $p\text{-value} < \alpha$.</li> For $\beta_1$ in SLR: $t = S_{xy}/\sqrt{MSE \cdot S_{xx}}$ with $n - 2$ degrees of freedom.</li> Testing $\beta_1 = 0$ is equivalent to testing $\rho = 0$.</li> Confidence intervals and hypothesis tests provide equivalent information.</li> Always consider practical significance alongside statistical significance.</li> </ul> In the next post, we will examine the ANOVA framework, which tests the overall significance of the regression model using the F-test. Confidence Intervals for Regression Coefficients 2026-02-13T00:00:00+00:00 This note shows how confidence intervals for regression coefficients are constructed and how to interpret them correctly. The core issue is avoiding binary thinking around single-point estimates. The objective is to connect standard errors, critical values, and interval interpretation in a reproducible way. In the previous post</a>, we derived the standard errors of the regression coefficients. Standard errors tell us about the precision of our estimates, but they do not directly give us a range of plausible values for the true parameters. Confidence intervals do exactly that. The Sampling Distribution of $\hat{\beta}_j$</h2> Under the normality assumption $\varepsilon_i \sim N(0, \sigma^2)$, the OLS estimators are also normally distributed: $$\hat{\beta}_j \sim N\left(\beta_j, \sigma^2 C_{jj}\right),$$ where $C_{jj}$ is the appropriate diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$. Standardizing gives: $$\frac{\hat{\beta}_j - \beta_j}{\sigma\sqrt{C_{jj}}} \sim N(0, 1).$$ However, $\sigma^2$ is unknown and we replace it with $MSE$. This introduces additional uncertainty, and the resulting distribution changes from a standard normal to a t-distribution: $$\frac{\hat{\beta}_j - \beta_j}{SE(\hat{\beta}_j)} \sim t_{n - p},$$ where $SE(\hat{\beta}_j) = \sqrt{MSE \cdot C_{jj}}$ and $n - p$ is the degrees of freedom. Confidence Interval Formula</h2> A $(1 - \alpha) \times 100%$ confidence interval for $\beta_j$ is: $$\hat{\beta}_j \pm t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_j),$$ where $t_{\alpha/2, n-p}$ is the critical value from the t-distribution with $n - p$ degrees of freedom such that $P(|T| > t_{\alpha/2, n-p}) = \alpha$. Equivalently, the interval is: $$\left[\hat{\beta}_j - t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_j), \quad \hat{\beta}_j + t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_j)\right].$$ Confidence Interval for $\beta_1$ (Simple Linear Regression)</h2> For the slope in simple linear regression, $n - p = n - 2$: $$\hat{\beta}_1 \pm t_{\alpha/2,, n-2} \cdot SE(\hat{\beta}_1),$$ where: $$SE(\hat{\beta}_1) = \sqrt{\frac{MSE}{S_{xx}}}.$$ Expanding fully: $$\hat{\beta}_1 \pm t_{\alpha/2,, n-2} \cdot \sqrt{\frac{SSE}{(n-2) \cdot S_{xx}}}.$$ This interval is centered at the OLS estimate $\hat{\beta}_1$ and its width depends on three factors: Confidence level ($1 - \alpha$): Higher confidence means a wider interval (larger $t_{\alpha/2}$).</li> Residual variability ($MSE$): More noise means a wider interval.</li> Predictor variability ($S_{xx}$): More spread in $x$ means a narrower interval.</li> </ol> Confidence Interval for $\beta_0$ (Simple Linear Regression)</h2> For the intercept: $$\hat{\beta}_0 \pm t_{\alpha/2,, n-2} \cdot SE(\hat{\beta}_0),$$ where: $$SE(\hat{\beta}_0) = \sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right)}.$$ The intercept's confidence interval is typically wider than the slope's because the variance of $\hat{\beta}_0$ includes an extra term involving $\bar{x}^2/S_{xx}$. When $\bar{x}$ is far from zero, the intercept estimate becomes less precise. General Confidence Interval for $\beta_i$ (Multiple Regression)</h2> For any coefficient $\beta_i$ in the multiple regression model: $$\hat{\beta}_i \pm t_{\alpha/2,, n-p} \cdot SE(\hat{\beta}_i),$$ where: $$SE(\hat{\beta}_i) = \sqrt{MSE \cdot C_{ii}}$$ and $C_{ii}$ is the $(i+1)$-th diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$. The degrees of freedom are $n - p$, reflecting that $p$ parameters have been estimated. As more predictors are included (larger $p$), fewer degrees of freedom remain, and the critical value $t_{\alpha/2, n-p}$ increases, widening the intervals. Interpretation of Confidence Intervals</h2> A 95% confidence interval for $\beta_1$ does not mean there is a 95% probability that $\beta_1$ lies within the interval. The true parameter $\beta_1$ is a fixed (though unknown) constant; it either lies in the interval or it does not. The correct interpretation is: if we were to repeat the experiment many times and construct a 95% confidence interval each time, approximately 95% of those intervals would contain the true value $\beta_1$. When the Interval Contains Zero</h2> If the confidence interval for $\beta_j$ includes zero, this suggests that the data are consistent with $\beta_j = 0$, meaning the predictor $x_j$ may not have a statistically significant linear effect on $y$ at the chosen confidence level. Conversely, if the interval does not include zero, we have evidence that the predictor has a significant effect. This connection between confidence intervals and hypothesis testing is explored in the next post. Confidence Interval for the Mean Response</h2> We can also construct a confidence interval for the mean response at a given value $x_0$. The estimated mean response is: $$\hat{y}_0 = \hat{\beta}_0 + \hat{\beta}_1 x_0.$$ The variance of this estimate is: $$\text{Var}(\hat{y}_0) = \sigma^2\left(\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}\right).$$ The confidence interval for the mean response is: $$\hat{y}_0 \pm t_{\alpha/2,, n-2} \cdot \sqrt{MSE\left(\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}\right)}.$$ Notice that this interval is narrowest when $x_0 = \bar{x}$ and grows wider as $x_0$ moves away from $\bar{x}$. This reflects the fact that we are most confident about predictions near the center of the data. Summary</h2> In this post, we constructed confidence intervals for regression coefficients: The pivotal quantity $(\hat{\beta}_j - \beta_j)/SE(\hat{\beta}_j) \sim t_{n-p}$ forms the basis for inference.</li> For $\beta_1$: $\hat{\beta}_1 \pm t_{\alpha/2, n-2} \cdot \sqrt{MSE/S_{xx}}$.</li> For $\beta_0$: $\hat{\beta}_0 \pm t_{\alpha/2, n-2} \cdot \sqrt{MSE(1/n + \bar{x}^2/S_{xx})}$.</li> For any $\beta_i$ in multiple regression: $\hat{\beta}_i \pm t_{\alpha/2, n-p} \cdot \sqrt{MSE \cdot C_{ii}}$.</li> Interval width depends on the confidence level, residual variability, predictor spread, and sample size.</li> </ul> In the next post, we will use the same t-distribution framework to perform formal hypothesis tests on the regression coefficients. Standard Error in Regression 2026-02-06T00:00:00+00:00 This note explains standard error in regression as a measure of uncertainty in coefficient estimates, not model quality by itself. The practical question is how noisy your estimate is if the data-generating process were repeated. I connect residual variance, sample size, and confidence in coefficients. Estimating the Error Variance</h2> Under the linear model $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ with $\varepsilon_i \sim N(0, \sigma^2)$, the error variance $\sigma^2$ is unknown and must be estimated from the data. The natural estimate is based on the residuals. The Mean Squared Error (MSE) is defined as: $$MSE = \frac{SSE}{n - 2} = \frac{\sum^n_{i=1}(y_i - \hat{y}_i)^2}{n - 2}.$$ The denominator is $n - 2$ (not $n$) because we lose two degrees of freedom for estimating $\hat{\beta}_0$ and $\hat{\beta}_1$. This correction ensures that MSE is an unbiased estimator of $\sigma^2$: $$E(MSE) = \sigma^2.$$ For multiple regression with $p$ parameters, the denominator becomes $n - p$: $$MSE = \frac{SSE}{n - p}.$$ Variance of the OLS Estimators</h2> Simple Linear Regression</h3> The OLS estimators are functions of the random variable $\mathbf{Y}$ (since $\mathbf{X}$ is treated as fixed). Their variances can be derived from the model assumptions. Variance of $\hat{\beta}_1$: Since $\hat{\beta}_1 = S_{xy}/S_{xx} = \sum^n_{i=1}c_i y_i$ where $c_i = (x_i - \bar{x})/S_{xx}$, and the $y_i$ are independent with variance $\sigma^2$: $$\text{Var}(\hat{\beta}_1) = \sum^n_{i=1}c_i^2 \text{Var}(y_i) = \sigma^2 \sum^n_{i=1}\frac{(x_i - \bar{x})^2}{S_{xx}^2} = \sigma^2 \cdot \frac{S_{xx}}{S_{xx}^2} = \frac{\sigma^2}{S_{xx}}.$$ Variance of $\hat{\beta}_0$: Since $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$: $$\text{Var}(\hat{\beta}_0) = \text{Var}(\bar{y}) + \bar{x}^2 \text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{n} + \bar{x}^2 \cdot \frac{\sigma^2}{S_{xx}} = \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right).$$ Note that $\text{Var}(\hat{\beta}_0)$ depends on the mean of $x$: the further $\bar{x}$ is from zero, the larger the variance of the intercept estimate. Standard Errors</h2> The standard error of an estimator is the square root of its estimated variance, obtained by replacing $\sigma^2$ with MSE. Standard error of $\hat{\beta}_1$: $$SE(\hat{\beta}_1) = \sqrt{\frac{MSE}{S_{xx}}}.$$ Standard error of $\hat{\beta}_0$: $$SE(\hat{\beta}_0) = \sqrt{MSE\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right)}.$$ These formulas show that the precision of our estimates improves (standard errors decrease) when: $MSE$ is small (the model fits the data well).</li> $S_{xx}$ is large (there is more spread in the $x$ values).</li> $n$ is large (we have more data).</li> </ul> General Case: Multiple Regression</h2> For the multiple regression model $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$, the variance-covariance matrix of $\hat{\boldsymbol{\beta}}$ is: $$\text{Var}(\hat{\boldsymbol{\beta}}) = \sigma^2(\mathbf{X}^T\mathbf{X})^{-1}.$$ The estimated variance-covariance matrix is: $$\widehat{\text{Var}}(\hat{\boldsymbol{\beta}}) = MSE \cdot (\mathbf{X}^T\mathbf{X})^{-1}.$$ Let $(\mathbf{X}^T\mathbf{X})^{-1} = \mathbf{C}$, and let $C_{jj}$ denote the $(j+1)$-th diagonal element of $\mathbf{C}$ (corresponding to $\hat{\beta}_j$). Then: $$\text{Var}(\hat{\beta}_j) = \sigma^2 C_{jj},$$ $$SE(\hat{\beta}_j) = \sqrt{MSE \cdot C_{jj}}.$$ Verification for Simple Linear Regression</h3> For simple linear regression, the design matrix is: $$\mathbf{X} = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}, \quad \mathbf{X}^T\mathbf{X} = \begin{pmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2 \end{pmatrix}.$$ The inverse is: $$(\mathbf{X}^T\mathbf{X})^{-1} = \frac{1}{n\sum x_i^2 - (\sum x_i)^2}\begin{pmatrix} \sum x_i^2 & -\sum x_i \\ -\sum x_i & n \end{pmatrix}.$$ Since $n\sum x_i^2 - (\sum x_i)^2 = n \cdot S_{xx}$, the diagonal elements are: $$C_{00} = \frac{\sum x_i^2}{n \cdot S_{xx}} = \frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}, \quad C_{11} = \frac{n}{n \cdot S_{xx}} = \frac{1}{S_{xx}}.$$ These give us: $$\text{Var}(\hat{\beta}_0) = \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right), \quad \text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{S_{xx}},$$ which matches our earlier derivations. Interpreting the Standard Error</h2> The standard error measures the typical size of the sampling variability in the estimated coefficient. If we were to repeatedly draw new samples of size $n$ from the same population and compute $\hat{\beta}_1$ each time, the standard deviation of these estimates would be approximately $SE(\hat{\beta}_1)$. A small standard error relative to the coefficient estimate suggests that the estimate is precise. A large standard error suggests substantial uncertainty. We will use the standard error to construct confidence intervals (Post 7) and perform hypothesis tests (Post 8). Summary</h2> In this post, we derived the standard errors of the regression coefficients: $MSE = SSE/(n-2)$ estimates the error variance $\sigma^2$ for simple linear regression, or $MSE = SSE/(n-p)$ for multiple regression.</li> $SE(\hat{\beta}_1) = \sqrt{MSE/S_{xx}}$ measures the precision of the slope estimate.</li> $SE(\hat{\beta}_0) = \sqrt{MSE(1/n + \bar{x}^2/S_{xx})}$ measures the precision of the intercept estimate.</li> For multiple regression, $SE(\hat{\beta}_j) = \sqrt{MSE \cdot C_{jj}}$ where $C_{jj}$ is the $(j+1)$-th diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</li> Precision improves with larger $S_{xx}$, smaller $MSE$, and larger $n$.</li> </ul> In the next post, we will use these standard errors to construct confidence intervals for $\beta_0$, $\beta_1$, and $\beta_i$. R-squared: SST, SSE, SSR and the Relationship with Correlation 2026-01-30T00:00:00+00:00 This note explains SST, SSR, SSE, and R-squared, then connects these quantities to correlation in the simple regression setting. The aim is to clarify what variance decomposition actually tells you about model fit. It also points out where high R-squared can still be misleading. In the previous post</a>, we introduced the Pearson correlation coefficient $r$. In this post, we define the coefficient of determination $R^2$ through the decomposition of total variability, and then prove that $R^2 = r^2$ for simple linear regression. The Sum of Squares Decomposition</h2> The total variability in the observed response values can be decomposed into two components: the variability explained by the regression model and the variability left unexplained (the residuals). Total Sum of Squares (SST)</h3> $$SST = \sum^n_{i=1}(y_i - \bar{y})^2 = S_{yy}.$$ This measures the total variability of $y$ around its mean. Sum of Squares due to Regression (SSR)</h3> $$SSR = \sum^n_{i=1}(\hat{y}_i - \bar{y})^2.$$ This measures the variability in $y$ that is explained by the regression model. Sum of Squared Errors (SSE)</h3> $$SSE = \sum^n_{i=1}(y_i - \hat{y}_i)^2.$$ This measures the variability in $y$ that remains unexplained. The Identity</h3> The fundamental decomposition is: $$SST = SSR + SSE.$$ To see why this holds, write: $$y_i - \bar{y} = (\hat{y}_i - \bar{y}) + (y_i - \hat{y}_i).$$ Squaring both sides and summing over all observations: $$\sum_{i=1}^n (y_i - \bar{y})^2 = \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i=1}^n (y_i - \hat{y}_i)^2 + 2\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i).$$ The cross term vanishes because of the properties of OLS residuals: $$\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) = \sum_{i=1}^n (\hat{y}_i - \bar{y}),e_i = 0.$$ This holds because OLS residuals are orthogonal to the fitted values. Therefore: $$SST = SSR + SSE.$$ The Coefficient of Determination $R^2$</h2> The coefficient of determination is defined as: $$R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}.$$ Both expressions are equivalent because $SSR = SST - SSE$. The value of $R^2$ represents the proportion of the total variability in $y$ that is explained by the regression model: $R^2 = 1$: the model explains all the variability (all data points lie exactly on the fitted line).</li> $R^2 = 0$: the model explains none of the variability (the fitted line is just $\hat{y} = \bar{y}$).</li> $0 \leq R^2 \leq 1$: in general, $R^2$ lies between 0 and 1 for models with an intercept.</li> </ul> Proof that $R^2 = r^2$ for Simple Linear Regression</h2> We now prove that for simple linear regression, the coefficient of determination equals the square of the Pearson correlation coefficient. Step 1: Express SSR in terms of $S_{xx}$, $S_{xy}$</h3> For simple linear regression, the fitted values are: $$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i = (\bar{y} - \hat{\beta}_1\bar{x}) + \hat{\beta}_1 x_i = \bar{y} + \hat{\beta}_1(x_i - \bar{x}).$$ Therefore: $$\hat{y}_i - \bar{y} = \hat{\beta}_1(x_i - \bar{x}).$$ Substituting into SSR: $$SSR = \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 = \hat{\beta}_1^2 \sum_{i=1}^n (x_i - \bar{x})^2 = \hat{\beta}_1^2 \cdot S_{xx}.$$ Since $\hat{\beta}_1 = S_{xy}/S_{xx}$: $$SSR = \left(\frac{S_{xy}}{S_{xx}}\right)^2 S_{xx} = \frac{S_{xy}^2}{S_{xx}}.$$ Step 2: Compute $R^2$</h3> $$R^2 = \frac{SSR}{SST} = \frac{S_{xy}^2 / S_{xx}}{S_{yy}} = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}}.$$ Step 3: Compare with $r^2$</h3> From the previous post</a>, the correlation coefficient is: $$r = \frac{S_{xy}}{\sqrt{S_{xx} \cdot S_{yy}}}.$$ Squaring both sides: $$r^2 = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}}.$$ Therefore: $$R^2 = r^2. \quad \blacksquare$$ This result is elegant and important. It tells us that for simple linear regression, the proportion of variance explained by the model is exactly the square of the correlation between $x$ and $y$. Alternative Proof via SSE</h2> We can also prove the result by expressing SSE in terms of $S_{xx}$, $S_{yy}$, $S_{xy}$. Since $e_i = y_i - \hat{y}_i = y_i - \bar{y} - \hat{\beta}_1(x_i - \bar{x})$: $$\begin{aligned} SSE &= \sum_{i=1}^n \big[(y_i - \bar{y}) - \hat{\beta}_1 (x_i - \bar{x})\big]^2 \\ &= \sum_{i=1}^n (y_i - \bar{y})^2 - 2\hat{\beta}_1 \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) + \hat{\beta}_1^2 \sum_{i=1}^n (x_i - \bar{x})^2 \\ &= S_{yy} - 2\hat{\beta}_1 S_{xy} + \hat{\beta}_1^2 S_{xx}. \end{aligned}$$ Substituting $\hat{\beta}_1 = S_{xy}/S_{xx}$: $$\begin{aligned} SSE &= S_{yy} - 2\cdot\frac{S_{xy}}{S_{xx}}\cdot S_{xy} + \frac{S_{xy}^2}{S_{xx}^2}\cdot S_{xx} \\ &= S_{yy} - \frac{2S_{xy}^2}{S_{xx}} + \frac{S_{xy}^2}{S_{xx}} \\ &= S_{yy} - \frac{S_{xy}^2}{S_{xx}}. \end{aligned}$$ Therefore: $$R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{S_{yy} - S_{xy}^2/S_{xx}}{S_{yy}} = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}} = r^2.$$ Adjusted $R^2$ for Multiple Regression</h2> For multiple regression with $p$ parameters, $R^2$ always increases (or stays the same) when additional predictors are added, even if those predictors have no real relationship with $y$. To account for this, the adjusted $R^2$ penalizes for the number of predictors: $$R^2_{\text{adj}} = 1 - \frac{SSE/(n - p)}{SST/(n - 1)} = 1 - \frac{n - 1}{n - p}(1 - R^2).$$ Unlike $R^2$, the adjusted version can decrease when an uninformative predictor is added, making it more suitable for model comparison. Note that for simple linear regression ($p = 2$), $R^2 = r^2$ still holds, and $R^2_{\text{adj}}$ simplifies accordingly. Where a High R-squared Can Mislead</h2> Everything above describes what $R^2$ measures. It is equally important to be clear about what it does not measure. First, $R^2$ quantifies fit, not correctness. It tells us how much of the variability in $y$ is captured by the fitted line, but it says nothing about whether a line was the right model to fit. A clearly nonlinear pattern can still produce a high $R^2$ if the linear component happens to be strong. Anscombe's quartet is the classic reference here: four datasets with essentially identical $R^2$ (and identical fitted lines), one of which is a clean linear relationship, one a perfect parabola, and two dominated by single outlying points. The residual plot distinguishes them; $R^2$ does not. Second, plain $R^2$ never decreases when a predictor is added. Since OLS minimizes SSE over a strictly larger space of models, SSE can only stay the same or shrink, so $R^2 = 1 - SSE/SST$ can only stay the same or grow, even when the new predictor is pure noise. This is exactly the problem the adjusted $R^2$ above is designed to correct. Third, a high $R^2$ is not evidence of causation, and it is not a measure of predictive performance on new data. It is computed on the same observations used to fit the model, so it says nothing about how the model generalizes out of sample. Summary</h2> In this post, we covered the sum of squares decomposition and the coefficient of determination: $SST = SSR + SSE$ decomposes total variability into explained and unexplained components.</li> $R^2 = SSR/SST = 1 - SSE/SST$ measures the proportion of variance explained.</li> For simple linear regression, $R^2 = r^2 = S_{xy}^2/(S_{xx} \cdot S_{yy})$, proven by expressing SSR in terms of $S_{xx}$ and $S_{xy}$.</li> Adjusted $R^2$ penalizes for model complexity in multiple regression.</li> </ul> In the next post, we will derive the standard errors of the regression coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$ using $S_{xx}$ and MSE. Correlation 2026-01-23T00:00:00+00:00 This note defines correlation clearly, explains how to compute it, and separates association from causation. The key issue is that correlation is often over-interpreted as evidence of mechanism. The goal is to make correlation a diagnostic tool, not a conclusion. In the previous posts, we built the regression framework using $S_{xx}$ and $S_{xy}$ to derive the OLS estimators. Before we move to deeper inferential topics, it is essential to formalize a measure of the strength of the linear association between two variables. This measure is the Pearson correlation coefficient. Recap of Key Notation</h2> In Simple Linear Regression</a>, we introduced: $$S_{xx} = \sum^n_{i=1}(x_i - \bar{x})^2,$$ $$S_{xy} = \sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y}).$$ We now introduce the remaining quantity: $$S_{yy} = \sum^n_{i=1}(y_i - \bar{y})^2.$$ These three summary statistics capture the variability of $x$, the variability of $y$, and their joint variability, respectively. Sample Standard Deviations</h2> The sample standard deviations are defined as: $$S_x = \sqrt{\frac{S_{xx}}{n - 1}} = \sqrt{\frac{1}{n-1}\sum^n_{i=1}(x_i - \bar{x})^2},$$ $$S_y = \sqrt{\frac{S_{yy}}{n - 1}} = \sqrt{\frac{1}{n-1}\sum^n_{i=1}(y_i - \bar{y})^2}.$$ Note that $S_x$ and $S_y$ are the standard deviations using the $n - 1$ denominator (Bessel's correction), which gives unbiased estimates of the population standard deviations $\sigma_x$ and $\sigma_y$. The sample variances are simply: $$S_x^2 = \frac{S_{xx}}{n-1}, \quad S_y^2 = \frac{S_{yy}}{n-1}.$$ The Pearson Correlation Coefficient</h2> The sample Pearson correlation coefficient is defined as: $$r = \frac{S_{xy}}{\sqrt{S_{xx} \cdot S_{yy}}}.$$ Equivalently, using the standard deviations: $$r = \frac{\sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y})}{(n-1) S_x S_y} = \frac{S_{xy}}{(n-1) S_x S_y}.$$ The first form (using $S_{xx}$, $S_{yy}$, $S_{xy}$) is often more convenient for algebraic manipulation, while the second form highlights that $r$ is a standardized measure of covariation. Properties of the Correlation Coefficient</h2> The Pearson correlation coefficient has several important properties: 1. Bounded between -1 and 1</h3> $$-1 \leq r \leq 1.$$ This follows from the Cauchy-Schwarz inequality, which states that $(S_{xy})^2 \leq S_{xx} \cdot S_{yy}$, with equality only when all data points lie exactly on a line. 2. Interpretation of values</h3> $r = 1$: perfect positive linear relationship (all points on a line with positive slope).</li> $r = -1$: perfect negative linear relationship (all points on a line with negative slope).</li> $r = 0$: no linear relationship (but there may be a nonlinear relationship).</li> $0 < |r| < 1$: partial linear association, with strength increasing as $|r|$ approaches 1.</li> </ul> 3. Symmetry</h3> $$r_{xy} = r_{yx}.$$ The correlation between $x$ and $y$ is the same as the correlation between $y$ and $x$. This follows because $S_{xy}$ is symmetric in $x$ and $y$. 4. Invariance under linear transformation</h3> If we define $u_i = a + bx_i$ and $v_i = c + dy_i$ with $b > 0$ and $d > 0$, then the correlation between $u$ and $v$ equals the correlation between $x$ and $y$. If either $b$ or $d$ is negative (but not both), the sign of $r$ flips. 5. Dimensionless</h3> The correlation coefficient has no units. The numerator $S_{xy}$ has units of $x$ times $y$, and the denominator $\sqrt{S_{xx} \cdot S_{yy}}$ also has units of $x$ times $y$, so they cancel. Relationship to the Regression Slope</h2> Recall from Post 1 that the OLS slope estimator is $\hat{\beta}_1 = S_{xy}/S_{xx}$. The correlation coefficient can be expressed in terms of $\hat{\beta}_1$: $$r = \hat{\beta}_1 \sqrt{\frac{S_{xx}}{S_{yy}}} = \hat{\beta}_1 \cdot \frac{S_x}{S_y}.$$ This shows that $r$ and $\hat{\beta}_1$ always share the same sign. A positive slope corresponds to a positive correlation, and a negative slope corresponds to a negative correlation. Conversely, we can write the slope as: $$\hat{\beta}_1 = r \cdot \frac{S_y}{S_x}.$$ This expresses the regression slope as the correlation times the ratio of the standard deviations. Population Correlation</h2> The sample correlation $r$ estimates the population correlation coefficient $\rho$ (rho), defined for a bivariate population as: $$\rho = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y} = \frac{E[(X - \mu_X)(Y - \mu_Y)]}{\sigma_X \sigma_Y}.$$ When the data $(x_i, y_i)$ are sampled from a bivariate normal distribution, $r$ is a consistent estimator of $\rho$. Correlation Does Not Imply Causation</h2> A strong correlation between two variables does not mean that one causes the other. The association might be due to a lurking variable, reverse causation, or coincidence. Regression models describe associations; establishing causation requires careful experimental design or additional assumptions. Summary</h2> In this post, we introduced the Pearson correlation coefficient: $S_{yy} = \sum(y_i - \bar{y})^2$ completes the set of summary statistics alongside $S_{xx}$ and $S_{xy}$.</li> The sample standard deviations $S_x = \sqrt{S_{xx}/(n-1)}$ and $S_y = \sqrt{S_{yy}/(n-1)}$ measure spread.</li> The correlation $r = S_{xy}/\sqrt{S_{xx} \cdot S_{yy}}$ quantifies the strength of the linear relationship.</li> $r$ is bounded between $-1$ and $1$, symmetric, and dimensionless.</li> The regression slope and correlation are related by $\hat{\beta}_1 = r \cdot S_y / S_x$.</li> </ul> In the next post, we will define $R^2$ using SST, SSE, and SSR, and prove mathematically that $R^2 = r^2$ for simple linear regression. Polynomial Regression 2026-01-16T00:00:00+00:00 This note explains polynomial regression as a linear model in transformed features, not a different estimation framework. The motivation is to capture nonlinear trends while retaining familiar regression machinery. I highlight when polynomial terms help and when they mainly increase variance. In the previous post</a>, we introduced multiple linear regression using matrix notation. One natural question arises: what if the relationship between $x$ and $y$ is not a straight line? Polynomial regression addresses this by fitting a polynomial function of a single variable $x$, and it turns out to be a special case of the multiple linear regression framework we have already developed. The Polynomial Model</h2> A polynomial regression model of degree $d$ takes the form: $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_d x_i^d + \varepsilon_i.$$ This model is nonlinear in the variable $x$, but it is still linear in the parameters $\beta_0, \beta_1, \ldots, \beta_d$. This distinction is important because it means we can use the same OLS estimation procedure from multiple linear regression. For example, a quadratic model ($d = 2$) is: $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \varepsilon_i,$$ and a cubic model ($d = 3$) is: $$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \varepsilon_i.$$ Design Matrix Construction</h2> To fit a polynomial regression using the matrix approach, we define new variables: $$z_1 = x, \quad z_2 = x^2, \quad z_3 = x^3, \quad \ldots, \quad z_d = x^d.$$ The design matrix for a polynomial of degree $d$ with $n$ observations is: $$\mathbf{X} = \begin{pmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^d \\ 1 & x_2 & x_2^2 & \cdots & x_2^d \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^d \end{pmatrix}.$$ This is exactly the design matrix for a multiple linear regression with $d$ predictors $z_1, z_2, \ldots, z_d$. Therefore, the OLS estimator is: $$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y},$$ which is the same formula we derived in the previous post. Why Polynomial Regression is a Special Case of MLR</h2> The key insight is that "linear" in "linear regression" refers to linearity in the parameters, not in the predictors. Although the polynomial model includes terms like $x^2$ and $x^3$, each coefficient $\beta_j$ appears linearly. We can treat each power of $x$ as a separate predictor variable, and the entire OLS theory from multiple linear regression applies directly. This means all the results we derived earlier carry over: The normal equations $\mathbf{X}^T\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{Y}$ hold.</li> The hat matrix $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ projects $\mathbf{Y}$ onto $\hat{\mathbf{Y}}$.</li> Residual properties remain the same.</li> </ul> Choosing the Degree</h2> A natural question is: what degree $d$ should we use? The choice involves a tradeoff between model flexibility and model complexity. Too low a degree (underfitting): The model is not flexible enough to capture the true relationship, leading to large systematic errors.</li> Too high a degree (overfitting): The model fits the training data very closely, including the noise, but performs poorly on new data.</li> </ul> A polynomial of degree $n - 1$ (where $n$ is the number of data points) can pass through every data point exactly, giving $SSE = 0$. However, such a model almost always overfits and generalizes badly. Common approaches to select the degree include: Visual inspection: Plot the data and the fitted curve for different degrees, and choose the one that captures the trend without fitting the noise.</li> Cross-validation: Split the data into training and validation sets, fit models of various degrees on the training set, and select the degree with the lowest prediction error on the validation set.</li> Information criteria: Use metrics such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) to balance goodness of fit with model complexity.</li> </ol> Multicollinearity Considerations</h2> One practical concern with polynomial regression is multicollinearity. The predictors $x, x^2, x^3, \ldots$ are often highly correlated, especially when $x$ values span a narrow range. High multicollinearity inflates the variance of the coefficient estimates and makes $\mathbf{X}^T\mathbf{X}$ nearly singular. A common remedy is to center the predictor before constructing polynomial terms. Instead of using $x$, we use $x - \bar{x}$: $$z_1 = x - \bar{x}, \quad z_2 = (x - \bar{x})^2, \quad z_3 = (x - \bar{x})^3, \quad \ldots$$ Centering reduces the correlation among the polynomial terms and improves the numerical stability of the estimation. Another approach is to use orthogonal polynomials, which are constructed so that $\mathbf{X}^T\mathbf{X}$ is diagonal, eliminating multicollinearity entirely. Example: Quadratic Fit</h2> Consider a quadratic model $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \varepsilon_i$ with the design matrix: $$\mathbf{X} = \begin{pmatrix} 1 & x_1 & x_1^2 \\ 1 & x_2 & x_2^2 \\ \vdots & \vdots & \vdots \\ 1 & x_n & x_n^2 \end{pmatrix}.$$ There are $p = 3$ parameters. The OLS solution $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ gives us $\hat{\beta}_0$, $\hat{\beta}_1$, and $\hat{\beta}_2$ simultaneously. The coefficient $\hat{\beta}_2$ indicates the curvature of the fitted parabola: a positive $\hat{\beta}_2$ means the curve opens upward, and a negative $\hat{\beta}_2$ means it opens downward. Summary</h2> In this post, we explored polynomial regression: The polynomial model $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_d x_i^d + \varepsilon_i$ is nonlinear in $x$ but linear in the parameters.</li> By treating each power of $x$ as a separate predictor, polynomial regression becomes a special case of multiple linear regression.</li> The OLS estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ applies directly.</li> Choosing the polynomial degree requires balancing fit and complexity to avoid overfitting.</li> Centering or using orthogonal polynomials helps address multicollinearity.</li> </ul> In the next post, we will examine correlation, which quantifies the strength of the linear relationship between two variables using $S_{xx}$, $S_{yy}$, and $S_{xy}$. Multiple Linear Regression 2026-01-09T00:00:00+00:00 This note extends linear regression to multiple predictors and explains how interpretation changes once features interact through a shared model. The practical challenge is understanding coefficients conditionally, not in isolation. I focus on the model form, estimation intuition, and common interpretation mistakes. In the previous post</a>, we explored simple linear regression with a single predictor. In practice, the response variable $y$ often depends on more than one predictor. Multiple linear regression extends the framework to accommodate $p - 1$ independent variables. The Model</h2> The multiple linear regression model for the $i$-th observation is: $$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_{p-1} x_{i,p-1} + \varepsilon_i,$$ where $x_{ij}$ is the value of the $j$-th predictor for the $i$-th observation, $\beta_j$ are the regression coefficients, and $\varepsilon_i$ is the error term. The total number of parameters is $p$ (including the intercept $\beta_0$). Matrix Notation</h2> Writing out the model for each observation individually becomes cumbersome as the number of predictors grows. Matrix notation provides a compact and powerful alternative. Define the following: $$\mathbf{Y} = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}, \quad \mathbf{X} = \begin{pmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1,p-1} \\ 1 & x_{21} & x_{22} & \cdots & x_{2,p-1} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{n,p-1} \end{pmatrix},$$ $$\boldsymbol{\beta} = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_{p-1} \end{pmatrix}, \quad \boldsymbol{\varepsilon} = \begin{pmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{pmatrix}.$$ The model can now be written as: $$\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}.$$ Here, $\mathbf{Y}$ is an $n \times 1$ vector of responses, $\mathbf{X}$ is an $n \times p$ design matrix (the first column of ones accounts for the intercept), $\boldsymbol{\beta}$ is a $p \times 1$ vector of coefficients, and $\boldsymbol{\varepsilon}$ is an $n \times 1$ vector of errors. OLS in Matrix Form</h2> The sum of squared errors can be written in matrix form as: $$SSE = (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})^T(\mathbf{Y} - \mathbf{X}\boldsymbol{\beta}).$$ Expanding this expression: $$SSE = \mathbf{Y}^T\mathbf{Y} - 2\boldsymbol{\beta}^T\mathbf{X}^T\mathbf{Y} + \boldsymbol{\beta}^T\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}.$$ To minimize, we take the derivative with respect to $\boldsymbol{\beta}$ and set it to zero: $$\frac{\partial SSE}{\partial \boldsymbol{\beta}} = -2\mathbf{X}^T\mathbf{Y} + 2\mathbf{X}^T\mathbf{X}\boldsymbol{\beta} = \mathbf{0}.$$ This gives us the normal equations in matrix form: $$\mathbf{X}^T\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{Y}.$$ Provided that $\mathbf{X}^T\mathbf{X}$ is invertible (that is, the columns of $\mathbf{X}$ are linearly independent), we can solve for the OLS estimator: $$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}.$$ This single formula generalizes the simple linear regression result. When $p = 2$ (one predictor plus the intercept), this reduces to $\hat{\beta}1 = S{xy}/S_{xx}$ and $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$ as derived in the previous post. Interpreting the Coefficients</h2> Each coefficient $\hat{\beta}_j$ (for $j = 1, 2, \ldots, p - 1$) represents the estimated change in $y$ for a one-unit increase in $x_j$, while holding all other predictors constant. This "holding other variables constant" interpretation is what distinguishes multiple regression from running separate simple regressions. The intercept $\hat{\beta}_0$ represents the estimated value of $y$ when all predictors are equal to zero. The Hat Matrix</h2> The vector of fitted values is: $$\hat{\mathbf{Y}} = \mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y} = \mathbf{H}\mathbf{Y},$$ where $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ is called the hat matrix. It "puts a hat on" $\mathbf{Y}$, transforming observed values into fitted values. The hat matrix is symmetric ($\mathbf{H}^T = \mathbf{H}$) and idempotent ($\mathbf{H}^2 = \mathbf{H}$). The residual vector is: $$\mathbf{e} = \mathbf{Y} - \hat{\mathbf{Y}} = (\mathbf{I} - \mathbf{H})\mathbf{Y}.$$ Assumptions</h2> The assumptions for multiple linear regression extend those of simple linear regression: Linearity: $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$, the model is linear in the parameters.</li> Full rank: The design matrix $\mathbf{X}$ has full column rank, meaning $\text{rank}(\mathbf{X}) = p$. This ensures $\mathbf{X}^T\mathbf{X}$ is invertible.</li> Exogeneity: $E(\boldsymbol{\varepsilon} | \mathbf{X}) = \mathbf{0}$, the errors have zero conditional mean.</li> Homoscedasticity: $\text{Var}(\boldsymbol{\varepsilon} | \mathbf{X}) = \sigma^2\mathbf{I}_n$, the errors have constant variance and are uncorrelated.</li> Normality (for inference): $\boldsymbol{\varepsilon} \sim N(\mathbf{0}, \sigma^2\mathbf{I}_n)$.</li> </ol> When assumptions 1 through 4 hold, the Gauss-Markov theorem guarantees that $\hat{\boldsymbol{\beta}}$ is the Best Linear Unbiased Estimator (BLUE). Connection to Simple Linear Regression</h2> In the special case where $p = 2$, we have a single predictor $x$ and the design matrix becomes: $$\mathbf{X} = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}.$$ Computing $(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ in this case yields the familiar results $\hat{\beta}1 = S{xy}/S_{xx}$ and $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$, confirming that the matrix formulation is a true generalization. Summary</h2> In this post, we extended the regression framework to handle multiple predictors: The model $\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$ uses matrix notation to express the relationship compactly.</li> The OLS estimator $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$ generalizes the simple linear regression solution.</li> Each $\hat{\beta}_j$ measures the effect of one predictor while holding the others constant.</li> The hat matrix $\mathbf{H}$ projects observed values onto fitted values.</li> </ul> In the next post, we will explore polynomial regression, which is a special case of multiple linear regression where the predictors are powers of a single variable. Simple Linear Regression 2026-01-02T00:00:00+00:00 This note introduces simple linear regression from first principles and focuses on how slope and intercept are estimated from data. The problem it solves is modeling a linear relationship between one predictor and one response. It is intended as the base layer for the later regression notes in this series. In secondary school, we learn that the equation of a straight line is given by $y = mx + c$, where $m$ is the slope and $c$ is the y-intercept. In statistics and machine learning, we use a similar but more general form to model the relationship between a dependent variable $y$ and an independent variable $x$. This is known as simple linear regression. The Model</h2> In simple linear regression, we express the relationship as: $$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i,$$ where $\hat{y}_i$ is the predicted value of the dependent variable for the $i$-th observation, $\hat{\beta}_0$ is the estimated y-intercept, and $\hat{\beta}_1$ is the estimated slope coefficient. The "hat" notation indicates that these are estimates derived from data, not the true (unknown) population parameters $\beta_0$ and $\beta_1$. The true model is assumed to be: $$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i,$$ where $\varepsilon_i$ represents the random error term for the $i$-th observation. Ordinary Least Squares (OLS)</h2> The question is: how do we find the best estimates $\hat{\beta}_0$ and $\hat{\beta}_1$? We need a systematic method that determines the line of best fit. The most common approach is Ordinary Least Squares (OLS). OLS minimizes the sum of the squared differences between the observed values $y_i$ and the predicted values $\hat{y}_i$. These differences are called residuals, defined as $e_i = y_i - \hat{y}_i$. By squaring the residuals, we treat positive and negative errors equally and penalize larger deviations more heavily. The objective function is the Sum of Squared Errors (SSE): $$SSE = \sum^n_{i=1}(y_i - \hat{y}_i)^2 = \sum^n_{i=1}(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2.$$ Deriving the Normal Equations</h2> To minimize the SSE, we take partial derivatives with respect to $\hat{\beta}_0$ and $\hat{\beta}_1$ and set them equal to zero. Partial derivative with respect to $\hat{\beta}_0$</h3> $$\frac{\partial SSE}{\partial \hat{\beta}_0} = -2\sum^n_{i=1}(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0.$$ Dividing both sides by $-2$ and expanding the sum: $$\sum^n_{i=1} y_i - n\hat{\beta}_0 - \hat{\beta}_1 \sum^n_{i=1} x_i = 0.$$ Solving for $\hat{\beta}_0$: $$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x},$$ where $\bar{x} = \frac{1}{n}\sum^n_{i=1}x_i$ and $\bar{y} = \frac{1}{n}\sum^n_{i=1}y_i$ are the sample means. Partial derivative with respect to $\hat{\beta}_1$</h3> $$\frac{\partial SSE}{\partial \hat{\beta}_1} = -2\sum^n_{i=1}x_i(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0.$$ Dividing by $-2$ and expanding: $$\sum^n_{i=1} x_i y_i - \hat{\beta}_0 \sum^n_{i=1} x_i - \hat{\beta}_1 \sum^n_{i=1} x_i^2 = 0.$$ Substituting $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$: $$\sum^n_{i=1} x_i y_i - (\bar{y} - \hat{\beta}_1 \bar{x})\sum^n_{i=1} x_i - \hat{\beta}_1 \sum^n_{i=1} x_i^2 = 0.$$ After simplification, we obtain: $$\hat{\beta}_1 = \frac{\sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y})}{\sum^n_{i=1}(x_i - \bar{x})^2}.$$ Introducing $S_{xx}$ and $S_{xy}$ Notation</h2> To write the estimators more concisely, we define the following summary statistics: $$S_{xx} = \sum^n_{i=1}(x_i - \bar{x})^2 = \sum^n_{i=1}x_i^2 - n\bar{x}^2,$$ $$S_{xy} = \sum^n_{i=1}(x_i - \bar{x})(y_i - \bar{y}) = \sum^n_{i=1}x_i y_i - n\bar{x}\bar{y}.$$ Using this notation, the OLS estimators become: $$\hat{\beta}_1 = \frac{S_{xy}}{S_{xx}},$$ $$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}.$$ These are elegant expressions that reveal the structure of the estimates. The slope $\hat{\beta}_1$ is the ratio of the joint variability of $x$ and $y$ (captured by $S_{xy}$) to the variability of $x$ alone (captured by $S_{xx}$). The intercept $\hat{\beta}_0$ ensures the regression line passes through the point $(\bar{x}, \bar{y})$. Residuals and Fitted Values</h2> Once we have $\hat{\beta}_0$ and $\hat{\beta}_1$, we can compute: Fitted values: $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ for each observation.</li> Residuals: $e_i = y_i - \hat{y}_i$ for each observation.</li> </ul> Two important properties of OLS residuals are worth noting: The residuals sum to zero: $\sum^n_{i=1} e_i = 0$.</li> The residuals are uncorrelated with the fitted values: $\sum^n_{i=1} e_i \hat{y}_i = 0$.</li> </ol> These properties follow directly from the normal equations. Key Assumptions</h2> For OLS to produce reliable estimates, the following assumptions are typically required: Linearity: The relationship between $x$ and $y$ is linear in the parameters.</li> Independence: The observations are independent of one another.</li> Homoscedasticity: The variance of the error terms $\varepsilon_i$ is constant across all values of $x$.</li> Normality: The error terms are normally distributed, that is, $\varepsilon_i \sim N(0, \sigma^2)$.</li> </ol> When these assumptions hold, OLS produces the Best Linear Unbiased Estimators (BLUE) according to the Gauss-Markov theorem. Summary</h2> In this post, we covered the fundamentals of simple linear regression: The model $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$ describes the estimated linear relationship between $x$ and $y$.</li> OLS minimizes the sum of squared errors to find the best-fitting line.</li> The estimators $\hat{\beta}1 = S{xy}/S_{xx}$ and $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$ are derived from the normal equations.</li> The $S_{xx}$ and $S_{xy}$ notation provides a compact way to express these results.</li> </ul> In the next post, we will extend this framework to handle multiple independent variables through multiple linear regression. Let's talk about Solar Photovoltaic Systems in WWTPs Malaysia 2025-12-01T00:00:00+00:00 This note reviews how solar photovoltaic systems can reduce energy cost and emissions in wastewater treatment plants (WWTPs) in Malaysia. The core problem is that WWTP operations are electricity-intensive and exposed to tariff volatility. I summarize constraints, control architecture, and where ML-based forecasting can improve deployment decisions. Why Solar PV in WWTPs?</h1> Wastewater treatment facilities are globally recognized as energy-intensive operations, often relying on electrical energy accounting for a significant portion of their operational expenditure (OPEX). For conventional WWTPs, energy use contributes between 25%-60% of the total OPEX ^{Muzaffar2022 . This high dependency on the central electricity grid exposes these facilities to electricity tariffs volatility and supply uncertainties. The national sewerage company, Indah Water Konsortium (IWK) Sdn Bhd, has experienced a substantial increase with electricity costs ballooned from RM22.53 million in 2000 to RM256.30 million in 2020 ^{IWK2021 . This became the primary motivation for IWK to explore renewable energy sources, particularly solar PV systems, to mitigate energy costs and enhance sustainability.}} IWK operates as Malaysia's national sewerage company and manages a vast network of public treatment plants. As of December 2021, IWK operated and maintained 7,272 public Sewerage Treatment Plants (STPs) and 1,375 network pump stations across the country ^{IWK2021_report . The regulatory oversight is provided by the Suruhanjaya Perkhidmatan Air Negara (SPAN), which ensures compliance with national water quality, ensuring that operators comply with stipulated standards and contractual obligations.} Energy Demand Characteristics of WWTPs</h1> Before implementing solar PV systems, it is crucial to understand the energy demand characteristics of WWTPs to correctly sizing and optimizing a solar PV system. The energy consumption profile of WWTP is highly influence by the pkant scale and the dominant treatment technology used. Specific Energy Consumption (SEC) and Categorization</h2> Energy intensity, generally measured as Specific Energy Consumption (SEC) in kilowatt-hour per cubic meter (kWh/m³) of treated wastewater, is a key metric for evaluating the energy efficiency of WWTPs. The SEC values can vary significantly based on the treatment processes employed and the plant's capacity. Data suggests that smaller WWTPs, particular those below $10,000 m^3/month$ capacity, tends to bear a disproportionately higher electricity cost, accounting for 30%-40% of their total running costs ^{Muzaffar2022 . In contrast, larger WWTPs with capacities exceeding typically exhibit lower SEC values, accounting for 15%-30% of the total running costs.} Specific data collected from village WWTPs in Romania indicated high annual average SEC values, ranging between $1.786 kWh/m^3$ to $2.334 kWh/m^3$ of treated wastewater ^{Tokos2021 . Furthermore, the complexity of sewage sludge disposal capacity has increased rapidly alongside population growthm reaching 7 millions $m^3$ annually. Managing this sludge requires significant energy input, contributing to the sector's overall energy consumption, with sewage sludge treatment alone consuming an estimated $544,900 GWh$ across IWK operations between 2016 and 2019 ^{Quan2022 .}} This observed variability indicates that the primary candidates for immediate solar PV intergartion are the small to medium sized conventional STPs. These plants have higher SEC values and higher dependence on expensive grid power, meaninf the marginal return from solar PV energy offset is maximized in this segment. Identification of Energy-Intensive Processes</h2> The greatest energy concentration of electrical energy demand within a conventional WWTP is typically found in the biological treatment line (BgT), primarily driven by the aeration systems. This process is necessary for activated sludge processes to maintain adequate dissolved oxygen levels for microbial activity. Data confirms that BgT accounts for the majority of total energy consumption, ranging from 63.2% to 72.9% in surveyed WWTPs ^{Tokos2021 .}Whether thorugh diffused air of mechanical aerators (such as those used in oxidation ditches</a>), aeration systems requires continuous operation to provide oxygen transfer, circulation, and mixing. This need for continuous, high-power during daytime operational hours, and we realized this characteristic is highly advantageous for solar PV integration. The steady daytime demand for blowers and aerators means that the generated solar power can be immediately and entirely consumed, leadning to high utilization rates, and diminish the need for Battery Energy Storage Systems (BESS), which is often a significant cost driver in renewable energy (RE) projects. The high concentration of energy use in aeration suggests that before implementing solar PV systems, WWTP operators should measures energy efficiency in the aeration systems. Simply installing a large PV array without optimzing the blowers and biological processes may lead to a larger and more costly system. Streamlining the anaerobic biological treatments is possible and should be prioritized to reduce the overall energy demand before sizing the solar PV system. Advanced PV Power Forecasting and Control Systems</h1> The modern integration of photovoltaic systems into WWTPs requires sophisticated forecasting and control architectures. Recent research demonstrates a clear evolution toward hybrid AI-based forecasting models that combine deep learning with optimization algorithms ^{IturraldeCarrera2025 . These advanced models significantly outperform classical deterministic methods in handling the highly dynamic and non-linear conditions of real-world PV generation.} Machine learning ensemble algorithms such as XGBoost, LightGBM, and CatBoost have emerged as foundational tools for high-accuracy solar power prediction ^{Nguyen2025 . A critical finding is that humidity and ambient temperature emerge as the most influential factors affecting PV module efficiency, particularly in tropical and humid climates.} Integrated Energy Management</h2> Effective PV system integration requires a multi-layered control hierarchy. The foundational layer is Maximum Power Point Tracking (MPPT), which continuously adjusts the DC-DC converter to extract maximum available power from the solar array. A critical technical constraint in many jurisdictions is the Zero Export requirement, which prohibits injecting electrical power back into the grid. To comply, Zero Export Controllers (ZEC) operate on near-instantaneous feedback loops ^{Alnawafah2025 .} Battery Energy Storage Systems provide flexibility to address both PV intermittency and zero-export constraints. Advanced control algorithms optimize charging during low-cost periods and discharging during peak tariff hours ^{Hvala2025 .} Conclusion</h1> The successful integration of solar PV systems into Malaysian WWTPs needs a strategic of advanced forecasting. By combining accurate generation forecasting with sophisticated demand-side prediction, facilities can achieve near energy-autonomous operation while positioning themselves as flexible power assets responding to grid demands. In the future research we need to explore the details implementation of predictive modelling and control algorithms in WWTPs, as well as conducting in-depth literature reviews on case studies of existing energy predictive systems in WWTP. Gradient Descent Algorithm Explained 2025-11-30T00:00:00+00:00 Gradient descent is simple to state and easy to get wrong in practice, almost always through the learning rate. This note works through the rule itself and then through what happens when the learning rate is too small, reasonable, and too large, with small examples you can follow by hand. The Update Rule</h2> Consider a differentiable function $f: \mathbb{R}^n \to \mathbb{R}$. The gradient of $f$ at a point $x \in \mathbb{R}^n$ is denoted as $\nabla f(x)$, which is a vector of partial derivatives. The gradient points in the direction of steepest ascent. To find a local minimum, gradient descent simply moves the other way: $$x_{k+1} = x_k - \alpha \nabla f(x_k),$$ where: $x_k$ is the current point,</li> $\alpha$ is the learning rate, where $\alpha > 0$,</li> $\nabla f(x_k)$ is the gradient of $f$ at point $x_k$.</li> </ul> We simulate two scenarios, one where the loss has a positive gradient and one where it is negative. Scenario 1: Positive Gradient</h3> Let's consider a simple quadratic function: $$f(x) = x^2 + 4x + 4.$$ The gradient of this function is: $$\nabla f(x) = 2x + 4.$$ Starting from an initial point, say $x_0 = 0$, and choosing a learning rate $\alpha = 0.1$, we can apply the gradient descent update rule iteratively: Compute the gradient at the current point: $\nabla f(x_0) = 2(0) + 4 = 4$.</li> Update the point: $x_1 = x_0 - 0.1 \cdot 4 = 0 - 0.4 = -0.4$.</li> Repeat the process for a number of iterations.</li> </ol> For the first 5 iteration, we can tabulate the results as follows: Iteration ($k$)</th> Current Point ($x_k$)</th> Gradient ($\nabla f(x_k)$)</th> Updated Point ($x_{k+1}$)</th></tr></thead> 0</td> 0.0</td> 4.0</td> -0.4</td></tr> 1</td> -0.4</td> 3.2</td> -0.72</td></tr> 2</td> -0.72</td> 2.56</td> -0.976</td></tr> 3</td> -0.976</td> 2.048</td> -1.1808</td></tr> 4</td> -1.1808</td> 1.6384</td> -1.34464</td></tr> </tbody></table> We can observe that the gradient os loss function from positive value is moving backward each time step, and the first step size is larger but it gradually decreases as we approach the minimum point. This is the brilliant part of gradient descent, as it automatically take larger steps when we are far from the minimum and smaller steps as we get closer to the minimum. We can visualize the process using a simple plot: import numpy as np import matplotlib.pyplot as plt # Define the function and its gradient def f(x): return x**2 + 4*x + 4 def grad_f(x): return 2*x + 4 # Gradient Descent parameters alpha = 0.1 x0 = 0 iterations = 20 # Store the points x_points = [x0] for _ in range(iterations): grad = grad_f(x_points[-1]) x_new = x_points[-1] - alpha * grad x_points.append(x_new) # Plotting x = np.linspace(-5, 1, 100) y = f(x) plt.plot(x, y, label='f(x) = x^2 + 4x + 4') plt.scatter(x_points, f(np.array(x_points)), color='red') plt.plot(x_points, f(np.array(x_points)), color='red', linestyle='--', label='Gradient Descent Path') plt.title('Gradient Descent on f(x)') plt.xlabel('x') plt.ylabel('f(x)') plt.legend() plt.grid() plt.show()</code></pre> After 20 iterations, the points converge towards the minimum point at $x = -2$. We can see how the points move along the curve of the function, gradually approaching the minimum. Scenario 2: Negative Gradient</h3> Now, let's consider a function with a negative gradient: $$f(x) = -x^3 + 4x^2 - 4.$$ The gradient of this function is: $$\nabla f(x) = -3x^2 + 8x.$$ Similar to what we have done on the previous example, we start from an initial point, say $x_0 = 0$, and choosing a learning rate $\alpha = 0.01$, we can apply the gradient descent update rule iteratively: Compute the gradient at the current point: $\nabla f(x_0) = -3(0)^2 + 8(0) = 0$.</li> Update the point: $x_1 = x_0 - 0.01 \cdot 0 = 0$.</li> Repeat the process for a number of iterations.</li> </ol> For the first 5 iterations, we can tabulate the results as follows: Iteration ($k$)</th> Current Point ($x_k$)</th> Gradient ($\nabla f(x_k)$)</th> Updated Point ($x_{k+1}$)</th></tr></thead> 0</td> 1.0000</td> 5.0000</td> 0.9500</td></tr> 1</td> 0.9500</td> 4.7175</td> 0.9028</td></tr> 2</td> 0.9028</td> 4.4533</td> 0.8583</td></tr> 3</td> 0.8583</td> 4.2057</td> 0.8162</td></tr> 4</td> 0.8162</td> 3.9734</td> 0.7765</td></tr> </tbody></table> In this scenario, we can observe that the gradient of loss function from negative value is moving forward each time step, and the first step size is larger but it gradually decreases as we approach the minimum point. Similar to the previous scenario, gradient descent automatically adjusts the step size based on the distance from the minimum. Similarly, we can visualize the process using a simple plot: import numpy as np import matplotlib.pyplot as plt # Define the function and its gradient def f(x): return -x**3 + 4*x**2 - 4 def grad_f(x): return -3*x**2 + 8*x # Gradient Descent parameters alpha = 0.01 x0 = 1 iterations = 40 # Store the points x_points = [x0] for _ in range(iterations): grad = grad_f(x_points[-1]) x_new = x_points[-1] - alpha * grad x_points.append(x_new) # Plotting x = np.linspace(-1, 3, 100) y = f(x) plt.plot(x, y, label='f(x) = -x^3 + 4x^2 - 4') plt.scatter(x_points, f(np.array(x_points)), color='red') plt.plot(x_points, f(np.array(x_points)), color='red', linestyle='--', label='Gradient Descent Path') plt.title('Gradient Descent on f(x)') plt.xlabel('x') plt.ylabel('f(x)') plt.legend() plt.grid() plt.show()</code></pre> After 20 iterations, the points converge towards the minimum point at approximately $x = 0$. We can see how the points move along the curve of the function, gradually approaching the minimum. We plotted the gradient descent path on the function curve: Proof of Convergence</h2> To prove the convergence of the gradient descent algorithm, we need to show that the sequence of points generated by the algorithm converges to a local minimum of the function $f(x)$. We assume that $f$ is a convex function with Lipschitz continuous gradients, meaning there exists a constant $L > 0$ such that for all $x, y \in \mathbb{R}^n$, $$|\nabla f(x) - \nabla f(y)| \leq L |x - y|.$$ Under convexity alone, we show that the rate of convergence is sublinear. Specifically, we can show that after $k$ iterations, the function value satisfies: $$ f(x_k) - f(x^*) \leq \frac{L |x_0-x^*|^2 }{2k}, $$ where $x^*$ is the global minimum point of $f$. This indicates that as the number of iterations $k$ increases, the function value approaches the minimum value at a rate inversely proportional to $k$. This completes the proof of convergence for the gradient descent algorithm under the assumptions of convexity and Lipschitz continuous gradients. The algorithm effectively finds a local minimum of the function $f(x)$ by iteratively updating the points in the direction of the steepest descent. But all of this hangs on the choice of learning rate $\alpha$. Too large and the algorithm overshoots the minimum and diverges. Too small and convergence crawls. There are various ways to adapt the learning rate during optimization, like schedules and optimizers such as Adam and RMSprop, which I may cover in a future post. Known Issues of Gradient Descent</h2> While gradient descent is a powerful optimization algorithm, it does have some known issues. In multivariate functions, the presence of saddle points can affect the convergence. Saddle points are points where the gradient is zero, but they are neither local minima nor local maxima. In high-dimensional spaces, saddle points are more prevalent than local minima, and gradient descent can get stuck at these points, leading to slow convergence or failure to find the global minimum. A popular example is the function $f(x, y) = x^2 - y^2$, which has a saddle point at $(0, 0)$. The direction vector at this point is zero, and gradient descent may struggle to escape this point. Gradient Descent Saddle Point</figcaption> </figure> To mitigate the issues with saddle points, various techniques can be employed, such as adding noise to the gradients, using momentum-based methods, or employing second-order optimization methods that consider the curvature of the function, which we can explore in future discussions. But overall, gradient descent remains a fundamental and widely used optimization algorithm in machine learning and various other fields. Conclusion</h2> That is gradient descent: a one-line update rule that ends up as the backbone of most of machine learning. It's remarkable how such a simple iterative process can lead optimize almost any complex function in real life applications. I truly cannot appreciate enough the beauty of this elegant mathematical concept. For those who are interested to learn more about gradient descent, I highly recommend watching the following video by StatQuest, which provides an excellent visual explanation of the algorithm: </iframe> </div> Gradient Descent, Step-by-Step | StatQuest</figcaption> </figure> I hope this post has provided a clear understanding of the gradient descent. My Gallery of Talentbank Boardroom Challenge 2025 2025-11-14T00:00:00+00:00 This post is a compact gallery plus debrief from the Talentbank Boardroom Challenge 2025. I keep the narrative focused on preparation, presentation decisions, and the specific lessons carried into later projects. </figure> </figure> </figure> </figure> </figure> </figure> Reinforcement learning practices in healthcare applications 2025-10-21T00:00:00+00:00 This note reviews practical reinforcement learning use cases in healthcare and the constraints that matter in deployment. The challenge is that policy learning in clinical settings is high-stakes, partially observed, and often offline. I summarize where RL is promising and where reliability and safety dominate design choices. In healthcare applications, artificial intelligence (AI) plays a crucial role in transforming patient care, diagnostics, and treatment planning to make healthcare more efficient and effective. However, if AI is used improperly, it may leads to worse outcomes rather than improved ones. In the subset of AI, reinforcement learning (RL) has shown great promise in optimising sequential decision-making processes, which are common settings in healthcare industry. However, applying RL in healthcare settings requires careful consideration of several important practices to ensure a safety outcome. To illustrate the pitfalls of reinforcement learning, we consider the sepsis management, which remains wide uncertainty in the way clinicians make decisions. In the context of sepsis, a history may include a patient's vital signs, laboratory results, administered treatments, and other relevant clinical information over time. The actions could involve decisions such as administering fluids, vasopressors, or antibiotics at different time points. The rewards are typically defined based on patient outcomes, such as survival rates, length of hospital stay, or improvement in clinical scores. Note that defining ideal sepsis resuscitation strategies is challenging due to the complex and dynamic nature of the condition, as well as the variability in patient responses to treatments, therefore it is not straightforward to define short-term rewards for each action taken. Here are three fundamental concerns when applying reinforcement learning in healthcare: Is the AI given access to all variables that influence decision making?</h1> RL agent can only ook at the recorded data, however there are much more information and context that should be taken into consideration. Failing to consider all variables may result in esitmates that are confounded by spurious correlation. For instance, severely sick septic patients may receive fluids earlier than healthier patients yet have worse outcomes, which is clearly because of them being sicker in the first place, not because of fluids worsen the outcomes. Therefore, it is important to considers of pissble confounding factors, which even more than what is required for standard prediction studies, as the sequential nature of the problem could possibly lead to confounding effects in both long term and short term. How big was that big data?</h1> This is relatively straightfoward. For any AI training, its necessary to feed the model an adequate amount of useful information, so is RL model. Logically, for RL model to evaluate a new policy, it needs to find a long, continuous sequence of decisions in the historical data that matches its new policy. In clinical trials, the mismatches between new treatment policy against historical data, also known as off-policy evaluation, the effective sample size can become small. The mismatches grow with the number of decisions in a patient's history. For one sepsis study, a cohort of 3,855 patients yielded an effective sample size of only a few dozen. Therefore, instead of exploring new treatment approaches, observational data shall be used only to refining existing practices. Will the AI behave prospectively as intended?</h1> One of the core elements in the feedback loop of every RL is rewards. However, if the design of reward function is not handled properly (e.g. error in formulation, data processing etc.), the momdel will eventually laed to poor decisions. Often, an overly simple reward function may neglect the long-term effects. For instance, rewarding only blood pressure targets may result in agent that harms long-term benefit by dosing excessive vasopressors to patients. Additionally, the learned policy might decay after a period of time if there's changes in the treatment standards. Therefore, it is neccessary to use interpretable machine learning to interrogate and assess whether learned policies will behave as intended in a prospective clinincal setting Conclusion</h1> In the end, although RL offers such promising opportunities to optimising sequential treatments in medical industry, we shall be cautious in deploying into production and requires due diligence to safely realise its potential in revolutionalising this life-saving industry. Guidelines for Reinforcement Learning in Healthcare</figcaption> </figure> In the end, I would like to express my gratitude to Omer Gottesman et al. for providing such practical viewpoint in standardising the application of reinforcement learning in clinical settings. References: Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., & Celi, L. A. (2019). Guidelines for reinforcement learning in healthcare. Nature Medicine, 25(1), 16–18. https://doi.org/10.1038/s41591-018-0310-5</a></li> </ul> Action-value methods with incremental step size in reinforcement learning 2025-10-17T00:00:00+00:00 This note derives the incremental update rule for action-value estimation in k-armed bandits and explains why it is preferable to recomputing full averages. The problem is memory and compute cost when rewards accumulate over time. By the end, you get a practical update equation you can implement directly in RL experiments. Consider any k-armed bandit problem</a>, we consider each actions taken as pulling an arm of a slot machine, and the machine gives us a reward based on the action taken. We denote the action selected at time step $t$ as $A_t$, and the correspondiong reward received as $R_t$. Among the $k$ actions, the expected value of each action $a$ is denoted as $q_*(a)=\mathbb{E}[R_t|A_t=a]$, which is also known as the value of that action $a$. It is rationale to choose the action with the highest value, however in the practical applications, the value of each action is unknown. Therefore, we need to estimate the value of each action, denoted as $Q_t(a)$, read as the estimated value of action $a$ at time step $t$. The fundamental goal is to find such $Q_t(a)$ that is as close as possible to the $q_*(a)$. It is commonly known as the true value by community. To estimate the values of each action, which we collectively call as the action-value methods, we can use the sample-average method to update the estimated value of action $a$ at time step $t$ as follows: $$Q_{t}(a)={{\text{sum of rewards when $a$ taken prior to $t$}} \over {\text{number of times $a$ taken prior to $t$}}}.$$ This method simply takes the average of all the rewards received when action $a$ is taken prior to time step $t$. To simplify the notation, we focus on a single action. We denoted $R_i$ as the reward received the $i$-th time of this action is taken, and $n$ as the number of times action $a$ is taken prior to time step $t$. Logically, we can let $Q_n$ denote the estimate of its action value after it has been taken $n-1$ times. Therefore, we can rewrite the update rule as follows: $$Q_n = \frac{\sum_{i=1}^{n-1}R_i}{n-1} = \frac{R_1+R_2+\ldots+R_{n-1}}{n-1}.$$ As the number of times of the action, $n$ increases, the obvious wat to update the estimate is to recalculate the average by summing up all the previous rewards and dividing it by $n-1$. However, as the number of times of the action increases to a large number, this method becomes progressively more expensive as we need to store all the previos rewards and recalculate the sum every time we need to update the estimate. But is there a better way to update the estimate without storing all the previous rewards? The answer is yes. We devise the incremental formulas for updating the estimate by $$ \begin{align*} Q_{n+1} & = \frac{1}{n}\sum_{i=1}^{n}R_i \\ & = \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1}R_i\right) \\ & = \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1}R_i\right) \\ & = \frac{1}{n}\left(R_n + (n-1)Q_n\right) \\ & = \frac{1}{n}\left(R_n+nQ_n-Q_n \right) \\ & = Q_n + \frac{1}{n}[R_n - Q_n]. \end{align*} $$ This incremental formula allows us to update the estimate $Q_n$ to $Q_{n+1}$ by only using the most recent reward $R_n$ and the previous estimate $Q_n$, without the need to store all the previous rewards. The term $\frac{1}{n}$ serves as the step size, which decreases as $n$ increases, ensuring that the estimate converges to the true value over time. Even in $n=1$, we can still obtain $Q_2 = R_1$ for arbitrary initial estimate $Q_1$. In this case, the initial estimate $Q_1$ is completely ignored after the first update, as it should be. In processing the $n$th reward, the estimate is adjusted by a fraction of the error term $[R_n - Q_n]$, which is the difference between the received reward and the current estimate. This adjustment is scaled by the step size $\frac{1}{n}$, which ensures that as more data is collected, the updates become smaller, allowing the estimate to stabilize around the true value. Note that the step size here is not constant, it decreases as the number of times of the action increases. Back to the bandit problem, the proposed simulation in for pseudo-code is as follows: Bandit Problem with Incremental Step Size</summary> Initialize Q(a) arbitrarily for all actions a For each time step t = 1, 2, ... Select action A_t using a policy derived from Q (e.g., ε-greedy) Take action A_t and observe reward R_t Update the estimate Q(A_t) using: n = number of times action A_t has been taken prior to time t Q(A_t) = Q(A_t) + (1/n) * [R_t - Q(A_t)] End For</code></pre> </div> </details> In the end of this post, we derived the incremental step size method for updating the aciton-value estimates which extensively applied in reinforcement learning. This method is computationally efficient as it does not require storing all previous rewards, and it ensures convergence to the true action values over time. Reinforcement Learning: An Introduction</figcaption> </figure> I want to express my gratitude to Sutton and Barto for their excellent book Reinforcement Learning: An Introduction</a> that provides a comprehensive introduction to the concepts and algorithms of reinforcement learning. References: Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf</a></li> </ul> A Small Talk About Hackathons 2025-08-07T00:00:00+00:00 This post is a short reflection on hackathon culture from a participant perspective: what helps teams learn fast, where teams usually waste time, and how to keep projects grounded under deadline pressure. Along the way, I've met people who are far more experienced, far more knowledgeable than me. And honestly, I can't compete with them. It's easy to feel small in those moments. For anyone who's been through hackathons, you'll know. The amount of time, energy, effort you need to commit is just INSANE. Most competitions require you to go through multiple stages: prelim round, and sometimes a semi-final, and then the final round. Every round is like a mini-marathon, endless brainstorming, last-minute changes, and almost no sleep just to push through and deliver something that works. In the first few hackathons, I was genuinely excited. It was fun. Every time one ended, I was already looking forward to the next. But after round and round of hackathons, I started to feel the burn. Exhausted, Emptiness. YES, I still learned something new in every game, but the energy, the excitement just started to wear off. Eventually, it all stated to feel a bit empty. There was also this lingering thought back in my mind: "Why am I comparing myself to people who are fully in this industry, who live and breathe in tech every single day, when I'm still just a student trying to explore things outside my field?" There's still one more hackathon coming up, and I'll give it what I can. but after that, I really need a break. A proper one. Not just from hackathons, but from the constant cycle of proving myself. I need to breathe, reset and work on my inner health, mentally and emotionally. Not saying I'm quitting hackathons forever. Not at all. But I know I need some time to focus more on myself, rebuild myself, and come back stronger. Thanks for reading until here. Sometimes, the best thing you can do for your growth is step back, realign, and then go again. Dead Internet Theory #1 2025-07-22T00:00:00+00:00 This post is a personal reflection on authenticity online: what feels different now, why AI-generated social content often feels hollow, and where I might be overreacting. The goal is not to claim a grand theory, but to document a concrete shift in reading experience across LinkedIn and Reddit. I do admit I used ChatGPT for my content in the past, but I realised that the content is not really what I’ve done before, the experience is basically “artificial”. I don’t deny AI as a productivity tool, but sometimes you just can’t tell the existence of the content written there. There’s a coldness, an emptiness that creeps in when scrolling through these posts, making it feel like shouting into a void where no real person listens. It is really different nowadays to scroll through social media, LinkedIn, and Reddit. You don’t feel people there. It makes me really feel like the dead Internet Theory is here, and everything is full of bot activity and automatically generated content manipulated by algorithms. The authenticity of online interactions seems to be fading, they replaced by automated perfection that feels disturbingly hollow. Chatbots or AI are really good for proofreading, but they are still only good at proofreading. Soon I think they will no longer have real content or real ideas when people are writing their experiences, their ideas; they just become full of BS nowadays. Now I even wish to find some long-ass article that is awfully organised, I find some fun to read through even though it was not good, but I can find authenticity there. Has anyone noticed this? You could share them with me or offer me a perspective I haven’t considered before. I’d be more than happy to discuss it. I think we’ll might have another follow-up episode on this. Share me your thoughts: contact@jienweng.com</a> Making Deepseek R1 ChatBot 2024-12-30T00:00:00+00:00 This post documents a small DeepSeek-R1 chatbot build: why I chose the model, what setup decisions mattered, and what worked in practice. Instead of focusing on AI industry drama, I keep the write-up centered on implementation choices and takeaways for future iterations. </a> But forget the drama for a second, because the best part? Deepseek is open-source. That’s a huge win for the AI community. No more being locked behind API paywalls or waiting for some corporate overlord to decide what we can or can’t do. It’s out there, free to tinker with, and you bet I had to try it out for myself. So, I went ahead and do something I wanted to do for soooo long -- built a chatbot. It’s not packed with fancy features (yet), but through this little experiment, I’ve discovered some pretty interesting things about how the Deepseek R1 model works. You can try it out live here</a>. btw, we won’t dive into the technical aspects just yet—that’s coming up in the next section! Stay tuned for more details on how these improvements will work behind the scenes. The Unique "Thinking" Approach</h3> What blows my mind the most about this whole setup is how I managed to separate the model’s thinking process from its final response. Most chatbots out there? They just spit out an answer, and you have no idea what’s happening behind the scenes. But with this, you can actually see how the model thinks through a problem before giving an answer. It’s like watching an AI have an inner monologue, refining its thoughts before speaking. And honestly? I’ve never seen this before in any LLMs I’ve used. At first, I didn’t even plan for this feature—it just happened while I was testing out different ways to improve response quality. I noticed that the model was generating some hidden reasoning steps before its final output. Instead of discarding them, I figured, Why not show them? And once I did, it was a game-changer. It made the AI feel so much more transparent—almost like it was thinking out loud. </a> For example, if you ask it something like, “What do you think about climate change in Malaysia?”, you won’t just get a final answer out of nowhere. You’ll actually see the model go through a step-by-step breakdown of its thought process: Breaking down the question components</li> Evaluating current knowledge</li> Forming logical connections</li> Synthesizing a comprehensive response</li> </ol> After seeing the model’s thinking process, what really stands out to me is how structured its response is. It doesn’t just throw out some generic take on climate change—it actually analyzes the question, breaks it down into different angles, and then builds a well-organized answer. </a> That said, while the response does sound solid, there are some oddities that make me wonder what’s going on under the hood. For example, it mentions “the subtropical Andaman and Nicobar Islands”—which, uh, aren’t even part of Malaysia. Also, “Ch bamboo” initiative? Never heard of that one. These small but noticeable mistakes show that while the model is good at structuring its answers, it still struggles with factual accuracy. But that’s exactly what makes having a visible thought process so useful. Instead of just blindly trusting AI responses, we can now see how the model arrives at its conclusions—which means we can spot errors more easily. If it had hallucinated this stuff in a normal chatbot, I might not have even noticed. But because I can watch it reason through the problem, I can tell where things might be going wrong. </a> This kind of transparency is what makes AI feel less like a magic black box and more like an actual tool that we can guide, correct, and refine. And that’s honestly what excites me the most about this project. Deployment Specifications</h3> The chatbot is currently hosted on Hugging Face Spaces, running on a basic-tier instance, which means it’s not exactly a powerhouse but still gets the job done. Here’s what it’s running on: CPU: 2 vCPUs</li> RAM: 16GB</li> Storage: Basic instance storage</li> Framework: Gradio</li> Inference Optimization: FP16 quantization</li> Average Response Time: 2-3 seconds</li> Concurrent Users Supported: Up to 10</li> </ul> You might notice that the live preview</a> here can be a bit slow while generating responses. That’s because the hardware isn’t optimized for LLM inference, so it’s working with some limitations. Hope you can bear with it! 😆 If you enjoy the project and want to see it run smoother, you can consider sponsoring me</a>. Who knows? With enough support, I might upgrade the resources for future projects and push this even further :D Efficient Model Architecture</h3> The chatbot uses the Deepseek R1 Distilled 1.5B model, which is a significantly compressed version of the original 685B parameter model. Despite having only 1.5 billion parameters, it maintains impressive performance for many tasks. Key points about the model: Original model: DeepSeek R1 (685B)</a></li> Distilled version: DeepSeek R1 Distill Qwen 1.5B</a></li> 440x parameter reduction while maintaining core capabilities</li> </ul> </a> Impressive Benchmark Results</h3> What’s most fascinating about this model is how well it holds up when compared to much larger models. Despite having far fewer parameters, it manages to outperform some big names in the AI world for certain tasks. </a> Outstanding Performance in Key Areas</h4> AIME 2024 (Math Competition) DeepSeek R1 Distilled: 28.9% Pass@1</li> GPT-4o: 9.3% Pass@1</li> Claude 3.5: 16.0% Pass@1</li> </ul> </li> MATH-500 (Mathematical Reasoning) DeepSeek R1 Distilled: 83.9% Pass@1</li> GPT-4o: 74.6% Pass@1</li> Claude 3.5: 78.3% Pass@1</li> </ul> </li> Codeforces (Competitive Programming) DeepSeek R1 Distilled: 954 Rating</li> GPT-4o: 759 Rating</li> Claude 3.5: 717 Rating</li> </ul> </li> </ol> Model Strengths & Limitations</h3> Strengths: Superior reasoning capabilities, especially in mathematics</li> Highly efficient with only 1.5B parameters</li> Effective knowledge distillation from larger models</li> Excellent performance in zero-shot scenarios</li> </ul> Limitations: Lower performance in general coding tasks</li> Potential language mixing issues</li> Sensitivity to prompt formatting</li> Limited performance in broader general knowledge tasks</li> </ul> This balanced perspective shows why I chose this model for my chatbot implementation - it provides exceptional reasoning capabilities while remaining lightweight enough for practical deployment. Try It Yourself</h3> Due to iframe restrictions, you can access the live demo through these methods: Direct Link to Demo</a></li> API Documentation</a></li> Source Code</a></li> </ol> </a> Summary</h3> Deepseek has definitely shaken things up in the AI world, and the drama surrounding it is just the tip of the iceberg. Forget the finger-pointing—this move is a win for the AI community, especially since Deepseek is open-source. No more waiting around for companies to decide how we can use AI; now it’s out there for everyone to play with and improve. And as for my little experiment—building a chatbot with the Deepseek R1 model—it’s not feature-packed yet, but it’s definitely been a fun ride. You can try it out live here</a> and see how it works for yourself! Additional Resources</h3> Model Card</a></li> Deployment Guide</a></li> Performance Benchmarks</a></li> Community Discussion</a></li> </ul> Feel free to experiment with the live demo and share your thoughts! References</h3> DeepSeek R1 (685B) The original DeepSeek R1 model, a large-scale AI model with 685 billion parameters, was the precursor to the distilled 1.5B version used in the chatbot. Source</a></li> DeepSeek R1 Distill Qwen 1.5B This is the distilled version of the DeepSeek R1 model, compressed to 1.5 billion parameters while retaining core capabilities. Source</a></li> Open R1 Model Architecture Explore the detailed architecture of the DeepSeek R1 model, showcasing its design and structure. Source</a></li> Medium - Deepseek R1 Distill Qwen 1.5B Performance A comparison of the performance between Deepseek R1 Distilled and other models, showing its impressive results in multiple domains. Source</a></li> Hugging Face Space - Chatbot Demo Live demo of the Deepseek R1 chatbot that showcases the model’s response and reasoning capabilities. Source</a></li> Hugging Face - API Documentation Official API documentation for Hugging Face Spaces, helping developers interact with models and integrate them into applications. Source</a></li> Hugging Face - Source Code Direct access to the source code of the Deepseek R1 chatbot project on Hugging Face Spaces for those interested in contributing or learning. Source</a></li> Hugging Face - Model Card Official card for the Deepseek R1 Distilled model, providing details on its functionality and training specifications. Source</a></li> Hugging Face - Deployment Guide Guidelines for deploying models and applications using Hugging Face Spaces. Source</a></li> Hugging Face - Performance Benchmarks An overview of the model performance across various tasks and benchmarks, showcasing the strengths and weaknesses of different models. Source</a></li> Hugging Face - Community Discussion Join the community discussions on Hugging Face, where users can ask questions, share insights, and discuss AI-related topics. Source</a></li> </ol> Eco Finance: A Sustainable Future Prototype 2024-12-29T00:00:00+00:00 This post focuses on the Eco Finance prototype itself: the problem we targeted, the product concept, and what we learned from turning an idea into a demo under hackathon constraints. It is written as a project debrief rather than an event recap. We participated in PayHack 2024, quite a big hackathon event with many talented individuals. Our project, Eco Finance, focuses on the implementation of a carbon tax expected to be released in 2026. Although it initially targets specific industries, it is crucial and trending to implement this for everyone in Malaysia. Read more about the 2026 carbon tax here. European countries are already implementing ESG-centric policies in various industries, such as the automobile industry. These are just preliminary steps; now we want to delve deeper into the subject. </a> Understanding the idea</h2> Do you know how much carbon footprint you generate from ordering a Shopee parcel? Or how much carbon you generate by driving to work instead of taking public transport? It's challenging for people to visualize their footprint. Hence, we're here to make this visible to people, increasing their awareness about this issue. According to Visa, 80% of Malaysians are aware of the environmental impact of consumption. With the release of Malaysia's largest payment gateway provider and the latest project linking banks. How it works</h2> The OpenFinance API can seamlessly integrate people's transaction details, allowing us to aggregate a person's transactions and their carbon footprints. The calculation would be the emission factor times the amount, giving us the carbon footprint from those transactions. For example, if the emission factor for a specific merchant category is 0.5 kg CO2 per RM, and a person spends RM 100, the carbon footprint would be 0.5 kg CO2/RM * 100 RM = 50 kg CO2. There's an established merchant category code (MCC) in transaction details, where we only need to fine-tune and investigate the actual emission factors for each merchant code. This could be done by collaborating with the Department of Statistics Malaysia (DOSM) to conduct surveys and research among Malaysians. In this project, we are using dummy variables based on this ideology only. Monetizing the Ecosystem</h2> We circulate this whole ecosystem by monetizing it. We extract carbon credits from them using the formula from WORLDMETER, where the baseline of average carbon emission per person in Malaysia is 8 tons per year. With that, we can pool up carbon credits, extracted from the surplus from the calculation. There's a proven market potential with Bursa Carbon Exchange (BCX), established on 9 Dec 2022, which is available to trade carbon credits in Malaysia. Then we sell the carbon credits to major companies in Malaysia like Petronas and Maybank to help them offset their carbon credits. Read more about BCX here</a>. From Petronas</a>, it's evident that their future plan aims for net-zero carbon emission by 2050. This proves there is a market in Malaysia, and more people will enter the market and participate. Encouraging Eco-Friendly Practices</h2> How do we encourage eco-friendly spending habits in Malaysia? We aim to attract more people, even those who are not initially interested in eco-friendly practices, to join us. We can make Malaysia greener and more sustainable, at least in the sense of ESG. We choose to reward users with healthy spending habits in terms of eco-friendly spending and reward them with cash for being environmentally friendly. Sustainable Business Model</h2> We can summarize the business circulation here and make it sustainable as well, where we can become self-sustaining: Collect carbon credits from users</li> Pool up carbon credits</li> Certify carbon credits with Bursa Carbon Exchange (BCX)</li> Sell carbon credits to companies who need them</li> Reward users to encourage eco-friendly spending habits.</li> </ol> Basically that's it. Project Random Thingy</h2> Now, it's about the random thingys of the project. We have the project hosted at this link. Although it is not fully complete, it serves as a prototype and is yet to be an MVP. Considering we had only 24 hours to complete this from idea to execution, I mean, it's good for a first-timer. Right...? You can access the hosted project prototype here</a>. Feel free to browse through it. If you have any questions, please email me, and I'll personally explain it to you. You can also see the admin page by accessing here</a> or by changing the /dashboard</code> to /admin</code>. It looks something like this: Feel free to browse through it. Wrap-Up</h2> In conclusion, Eco Finance aims to make carbon footprints visible to individuals, encouraging eco-friendly spending habits and contributing to a greener Malaysia. By monetizing carbon credits and rewarding users, we create a sustainable ecosystem that benefits both the environment and the economy. It's really kesian that we couldn't make it to final though T.T Hope you guys like it :D Oh btw! Once again.. You can read the full story here</a>. First Physical Hackathon Experience 2024-12-02T00:00:00+00:00 This post records what our first physical hackathon taught us as a math-heavy team entering a software-first environment. I focus on concrete lessons from ideation, mentoring, and pitching that we can reuse in future competitions. I found myself staring at my phone, thumb hovering over the share button. "Am I really qualified for this?" I thought to myself. "What if we make fools of ourselves?" The doubts crept in like unwanted guests. But then another voice, stronger and more determined, pushed back: "When else will we get a chance like this? We might not be coders, but we know how to solve problems. Isn't that what hackathons are really about?" I reached out to my fellow mathematics coursemates: Janice, Roius, and Gwyn. "Hey, want to do something crazy?" I asked, half expecting them to laugh it off. To my surprise, their responses came quickly, filled with enthusiasm despite (or maybe because of) our collective inexperience. Only 2 of us had ever participated in a hackathon before, and Gwyn had never even written a line of code. But there we were, four mathematics students from UTAR, signing up for one of the most competitive hackathons in the country. The looks we got when we arrived were priceless. "Are you guys from Computer Science?" someone asked, eyeing our team with curiosity. We exchanged glances and grinned. "Nope, we're Math students, haha..." The mixture of surprise and skepticism on their faces was something I'll never forget. In those moments, our outsider status felt both terrifying and weirdly empowering. November 30th marked our first day, and it was a blur of ideation and learning. We came up with an innovative idea: creating an app to visualize transaction carbon footprints, with the ability to pool and trade carbon footprint surpluses with companies in need. On paper, it sounded promising—a perfect blend of fintech and environmental consciousness. Then came the intense mentoring sessions with Johan Nasir. He didn't hold back. "What happens if the carbon credits are manipulated?" he'd challenge. "How do you ensure the authenticity of the footprint data?" Another round of rethinking. Each session felt like an intense rotan session—tough love at its finest. He'd poke holes in our solutions, push us to think deeper, and force us to confront real-world problems that actually needed solving. The week before the hackathon was a rollercoaster. Competing against almost 100 teams from across Malaysia, we were shocked to make it to the Top 32. When we saw Johan's name among our judges and received the news of our advancement, our excitement was through the roof. But reality quickly set in—we had a major problem. None of us had real web development experience. My knowledge was limited to Python, SQL, and vanilla HTML/CSS/JavaScript. With just four days between Tuesday and Friday, we had to make crucial technical decisions while juggling our internships. After intense research, we settled on Vue.js and Flask. Janice even traveled all the way from JB to KL for this. Every evening after our internships, we'd dive into tutorials, trying to absorb as much as we could about our chosen tech stack. The hackathon itself was intense. When exhaustion hit, we took turns napping wherever we could. I couldn't get a tent, so I made do with a random bench for quick 4-hour power naps before jumping back into coding. Everything seemed to be going smoothly until the morning of the submission. At 8:30 AM, just after breakfast and 90 minutes before the deadline, our backend crashed—the information couldn't be parsed properly. In a desperate move, we had to hardcode some components just to make the submission deadline at 10 AM. While we didn't make it to the Top 10, watching the final pitches was an eye-opening experience. The winning teams showcased solutions that were not just technically impressive but also deeply thoughtful about real-world implementation. One team's blockchain-based remittance system particularly stood out - their attention to regulatory compliance and market research was incredible. "We should have done more market validation," I thought to myself. "Next time, we need to focus not just on the technical solution but on the whole business case." The top teams also demonstrated masterful presentation skills. Their pitches weren't just about features - they told compelling stories about why their solutions mattered. Each slide was carefully crafted, each demo was flawlessly executed, and their responses to judges' questions showed deep understanding of both technical and business aspects. I made mental notes: "Practice the pitch more. Know your numbers. Be ready for any question." Despite not making the finals, I did win a Samsung monitor in the lucky draw, a small consolation that brought some laughs to our tired team. Looking back now, the sleepless nights and endless debugging sessions blur together, but certain moments stand crystal clear: the late-night breakthrough when our first feature finally worked, the proud smile on Johan's face during our final presentation, and most importantly, the unshakeable bond formed between four mathematicians who dared to dream. To Janice, Roius, and Gwyn: thank you for taking this leap of faith with me. For believing that our mathematical minds could contribute something meaningful to the tech world. To Johan: your guidance went beyond mentorship—you showed us that innovation comes from daring to be different. And to PayNet and JomHack: thank you for creating a space where even mathematics students could discover their potential in technology. They say the best stories come from stepping out of your comfort zone. Well, we didn't just step—we took a giant leap. And while our first hackathon journey has ended, something tells me this is just the beginning of our adventure in the tech world. The equations and formulas we've studied for years are no longer just abstract concepts—they're tools waiting to be applied in the vast playground of technology. After all, who says mathematicians can't be hackers too? Sometimes the best innovations come from those who dare to cross the boundaries between disciplines, who bring fresh perspectives to old problems. And maybe, just maybe, that's exactly what the tech world needs more of.

Jien Weng

The Fokker-Planck Equation Explained, from an SDE to the Density it Evolves

An Introduction to MLOps: A Complete Tutorial with a Hands-On

4. Make a second run to compare against</h3> Change one line, C = 0.01</code>, and run python train.py</code> again. (Stronger regularisation; the score should move.) Now you have two runs to compare</strong>, which is the whole point of tracking.</p>

Tree-Based Models Explained: From Decision Trees to Random Forest and XGBoost

Fast Fourier Transform Explained from the DFT, with a Sound-Wave Example

Softmax Regression Explained from Logistic Regression

Statistical Testing in Regression

For Multiple Regression</h3> Testing $H_0: \beta_i = 0$ for any coefficient $\beta_i$:</p> $$t = \frac{\hat{\beta}_i}{SE(\hat{\beta}_i)} = \frac{\hat{\beta}_i}{\sqrt{MSE \cdot C_{ii}}},$$</p> where $C_{ii}$ is the appropriate diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</p>

Testing Against a Non-Zero Value</h2> More generally, we can test $H_0: \beta_j = \beta_{j,0}$ for any hypothesized value $\beta_{j,0}$:</p> $$t = \frac{\hat{\beta}_j - \beta_{j,0}}{SE(\hat{\beta}_j)}.$$</p> The test for $\beta_j = 0$ is simply the special case where $\beta_{j,0} = 0$.</p>

Confidence Intervals for Regression Coefficients

Standard Error in Regression

Variance of the OLS Estimators</h2>

R-squared: SST, SSE, SSR and the Relationship with Correlation

Total Sum of Squares (SST)</h3> $$SST = \sum^n_{i=1}(y_i - \bar{y})^2 = S_{yy}.$$</p> This measures the total variability of $y$ around its mean.</p>

Sum of Squares due to Regression (SSR)</h3> $$SSR = \sum^n_{i=1}(\hat{y}_i - \bar{y})^2.$$</p> This measures the variability in $y$ that is explained by the regression model.</p>

Sum of Squared Errors (SSE)</h3> $$SSE = \sum^n_{i=1}(y_i - \hat{y}_i)^2.$$</p> This measures the variability in $y$ that remains unexplained.</p>

Proof that $R^2 = r^2$ for Simple Linear Regression</h2> We now prove that for simple linear regression, the coefficient of determination equals the square of the Pearson correlation coefficient.</p>

Step 2: Compute $R^2$</h3> $$R^2 = \frac{SSR}{SST} = \frac{S_{xy}^2 / S_{xx}}{S_{yy}} = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}}.$$</p>

Correlation

Properties of the Correlation Coefficient</h2> The Pearson correlation coefficient has several important properties:</p>

1. Bounded between -1 and 1</h3> $$-1 \leq r \leq 1.$$</p> This follows from the Cauchy-Schwarz inequality, which states that $(S_{xy})^2 \leq S_{xx} \cdot S_{yy}$, with equality only when all data points lie exactly on a line.</p>

3. Symmetry</h3> $$r_{xy} = r_{yx}.$$</p> The correlation between $x$ and $y$ is the same as the correlation between $y$ and $x$. This follows because $S_{xy}$ is symmetric in $x$ and $y$.</p>

4. Invariance under linear transformation</h3> If we define $u_i = a + bx_i$ and $v_i = c + dy_i$ with $b > 0$ and $d > 0$, then the correlation between $u$ and $v$ equals the correlation between $x$ and $y$. If either $b$ or $d$ is negative (but not both), the sign of $r$ flips.</p>

5. Dimensionless</h3> The correlation coefficient has no units. The numerator $S_{xy}$ has units of $x$ times $y$, and the denominator $\sqrt{S_{xx} \cdot S_{yy}}$ also has units of $x$ times $y$, so they cancel.</p>

Polynomial Regression

4. Make a second run to compare against</h3>
Change one line, `C = 0.01</code>, and run python train.py</code> again. (Stronger regularisation; the score should move.) Now you have two runs to compare</strong>, which is the whole point of tracking.</p>`

For Multiple Regression</h3>
Testing $H_0: \beta_i = 0$ for any coefficient $\beta_i$:</p>
$$t = \frac{\hat{\beta}_i}{SE(\hat{\beta}_i)} = \frac{\hat{\beta}_i}{\sqrt{MSE \cdot C_{ii}}},$$</p>
where $C_{ii}$ is the appropriate diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$.</p>

Testing Against a Non-Zero Value</h2>
More generally, we can test $H_0: \beta_j = \beta_{j,0}$ for any hypothesized value $\beta_{j,0}$:</p>
$$t = \frac{\hat{\beta}_j - \beta_{j,0}}{SE(\hat{\beta}_j)}.$$</p>
The test for $\beta_j = 0$ is simply the special case where $\beta_{j,0} = 0$.</p>

Total Sum of Squares (SST)</h3>
$$SST = \sum^n_{i=1}(y_i - \bar{y})^2 = S_{yy}.$$</p>
This measures the total variability of $y$ around its mean.</p>

Sum of Squares due to Regression (SSR)</h3>
$$SSR = \sum^n_{i=1}(\hat{y}_i - \bar{y})^2.$$</p>
This measures the variability in $y$ that is explained by the regression model.</p>

Sum of Squared Errors (SSE)</h3>
$$SSE = \sum^n_{i=1}(y_i - \hat{y}_i)^2.$$</p>
This measures the variability in $y$ that remains unexplained.</p>

Proof that $R^2 = r^2$ for Simple Linear Regression</h2>
We now prove that for simple linear regression, the coefficient of determination equals the square of the Pearson correlation coefficient.</p>

Step 2: Compute $R^2$</h3>
$$R^2 = \frac{SSR}{SST} = \frac{S_{xy}^2 / S_{xx}}{S_{yy}} = \frac{S_{xy}^2}{S_{xx} \cdot S_{yy}}.$$</p>

Properties of the Correlation Coefficient</h2>
The Pearson correlation coefficient has several important properties:</p>

1. Bounded between -1 and 1</h3>
$$-1 \leq r \leq 1.$$</p>
This follows from the Cauchy-Schwarz inequality, which states that $(S_{xy})^2 \leq S_{xx} \cdot S_{yy}$, with equality only when all data points lie exactly on a line.</p>

3. Symmetry</h3>
$$r_{xy} = r_{yx}.$$</p>
The correlation between $x$ and $y$ is the same as the correlation between $y$ and $x$. This follows because $S_{xy}$ is symmetric in $x$ and $y$.</p>

4. Invariance under linear transformation</h3>
If we define $u_i = a + bx_i$ and $v_i = c + dy_i$ with $b > 0$ and $d > 0$, then the correlation between $u$ and $v$ equals the correlation between $x$ and $y$. If either $b$ or $d$ is negative (but not both), the sign of $r$ flips.</p>

5. Dimensionless</h3>
The correlation coefficient has no units. The numerator $S_{xy}$ has units of $x$ times $y$, and the denominator $\sqrt{S_{xx} \cdot S_{yy}}$ also has units of $x$ times $y$, so they cancel.</p>