Underfitting is when a straight line fitted on the data causing the model to have high bias, unable to fit the training set well.

Overfitting: If we have too many features, the learned hypothesis may fit the training set very well, but fail to generalize to new examples or the test data. This is also referred to as a model with high variance. This can sometimes can occur when we have too many features in the training dataset.

Smaller values to the parameters produces smoother and simpler hypothesis curve, which is desired when having high-order degree polynomial hypothesis. This enable for less over-fitting.

Simple logistic regression is only able to find a linear decision boundary. For such cases where the data needs a non-linear hypothesis curve, one way to solve this is to use higher order polynomial features with regularization.

It is important to note that when implementing regularization, the first parameter is left untouched as modifying it might introduce some problem when creating the decision boundary.