CSC4030: Introduction to Machine Learning

Supervised learning trains a model on labeled data, where each training example includes both the input features and the correct output (label). The model learns a mapping from inputs to outputs and generalizes that mapping to predict labels for new, unseen inputs. Examples include predicting house prices from square footage and location (regression) or classifying emails as spam or not spam (classification). Unsupervised learning works with unlabeled data: the model receives only input features and must discover structure on its own. Common tasks include clustering (grouping similar customers for market segmentation), dimensionality reduction (compressing high-dimensional data for visualization or preprocessing), and anomaly detection (identifying unusual transactions that might indicate fraud). The practical difference is data availability: supervised learning requires someone to label the training data, which is expensive and time-consuming for large datasets. Unsupervised learning works with raw data but cannot make predictions in the same way. Many real-world ML systems combine both: use unsupervised methods to discover structure and create features, then feed those features into a supervised model for prediction.

Overfitting occurs when a model learns the training data too well, capturing noise and random fluctuations rather than the underlying pattern. An overfit model achieves high accuracy on training data but performs poorly on new, unseen data because it has memorized specific training examples rather than learning generalizable rules. Think of a student who memorizes every practice exam answer but cannot solve a slightly different problem on the real exam. Preventing overfitting involves several strategies: using more training data (the most effective remedy when feasible), applying regularization (L1/Lasso adds a penalty proportional to the absolute value of weights, pushing some to zero; L2/Ridge adds a penalty proportional to the square of weights, shrinking all weights toward zero), using dropout in neural networks (randomly deactivating neurons during training to prevent co-adaptation), applying early stopping (monitoring validation loss during training and stopping when it begins to increase even as training loss continues to decrease), and using cross-validation to get a more reliable estimate of model performance. The bias-variance trade-off is the underlying framework: overfitting means high variance (model is too sensitive to training data), underfitting means high bias (model is too simple to capture the pattern), and the goal is to find the right model complexity that minimizes total error.

Backpropagation is the algorithm that computes how much each weight in a neural network contributed to the prediction error, so that gradient descent can adjust the weights to reduce that error. It works in two phases. In the forward pass, input data flows through the network layer by layer: each neuron computes a weighted sum of its inputs, adds a bias, applies an activation function, and passes the result to the next layer. At the output layer, the loss function compares the network's prediction to the true label and produces a single error value. In the backward pass, backpropagation applies the chain rule of calculus to compute the gradient (partial derivative) of the loss with respect to each weight in the network, starting from the output layer and working backward to the input layer. Each weight's gradient tells you how much a small change in that weight would change the loss. The optimizer (SGD, Adam, or another variant) then updates each weight by subtracting a fraction (the learning rate) of its gradient, nudging the network toward lower loss. This process repeats for many iterations (epochs) over the training data. The key insight is that the chain rule makes gradient computation tractable even in deep networks with millions of parameters, because each layer's gradient depends only on the layer above it.

A simple train/test split divides data into two parts: train the model on one, evaluate on the other. The problem is that a single split can be misleading. If the test set happens to contain easy examples, the model looks better than it is; if it contains hard examples, the model looks worse. The performance estimate has high variance depending on which examples end up in which set. K-fold cross-validation addresses this by dividing the data into k equal parts (folds), training the model k times (each time using k-1 folds for training and 1 fold for testing), and averaging the k performance scores. This produces a more reliable estimate because every example serves as both training and test data across the k iterations. Common choices are k=5 or k=10. Stratified k-fold ensures that each fold preserves the class distribution of the full dataset, which is important for imbalanced datasets where one class is rare. Leave-one-out cross-validation (k equals the number of samples) provides the lowest bias estimate but is computationally expensive for large datasets. In CSC4030, cross-validation is the standard method for model comparison: when deciding between a random forest and a gradient boosting model, you compare their cross-validated scores rather than single train/test scores to make a more trustworthy decision.