Remove the outliers. However, do this if you are certain that the outliers are true anomalies not worth adding to your model. This should be your last consideration since dropping them means losing information.