Title: | Linear Ridge Regression with Ridge Penalty and Ridge Statistics |
---|---|
Description: | Linear ridge regression coefficient's estimation and testing with different ridge related measures such as MSE, R-squared etc. REFERENCES i. Hoerl and Kennard (1970) <doi:10.1080/00401706.1970.10488634>, ii. Halawa and El-Bassiouni (2000) <doi:10.1080/00949650008812006>, iii. Imdadullah, Aslam, and Saima (2017), iv. Marquardt (1970) <doi:10.2307/1267205>. |
Authors: | Imdad Ullah Muhammad [aut, cre]
|
Maintainer: | Imdad Ullah Muhammad <[email protected]> |
License: | GPL (>= 2.0) |
Version: | 1.2.2 |
Built: | 2025-02-18 05:25:53 UTC |
Source: | https://github.com/cran/lmridge |
R package for fitting linear ridge regression models.
This package contains functions for fitting linear ridge regression models, including functions for computation of different ridge related statistics (such as MSE, Var-Cov matrix, effective degrees of freedom and condition numbers), estimation of biasing parameter from different researchers, testing of ridge coefficients, model selection criteria, residuals, predicted values and fitted values. The package also includes function for plotting of ridge coefficients and different ridge statistics for selection of optimal value of biasing parameters .
For a complete list of functions, use library(help="lmridge")
.
Muhammad Imdad Ullah, Muhammad Aslam
Trade-off between bias, variance and MSE of the linear ridge regression against vector or scalar value of biasing parameter (see Kalivas and Palmer, 2014 <doi:10.1002/cem.2555>).
bias.plot(x, abline = TRUE, ...)
bias.plot(x, abline = TRUE, ...)
x |
An object of class "lmridge". |
abline |
Horizontal and vertical lines show the minimum value of the ridge MSE at certain value of biasing parameter |
... |
Not presently used in this implementation. |
The effect of multicollinearity on the coefficient estimates can be identified using different graphical display. One of them is plot of bias, variance and MSE. A little addition of bias lead to a substantial decrease in variance, and MSE. Therefore, a trade-off is made between bias and variance to have acceptable MSE. The bias.plot
can be helpful for selection of optimal value of biasing parameter .
Nothing returned
Muhammad Imdad Ullah, Muhammad Aslam
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Kalivas, J. H., and Palmer, J. (2014). Characterizing multivariate calibration tradeoffs (bias, variance, selectivity, and sensitivity) to select model tuning parameters. Journal of Chemometrics, 28(5), 347–357. doi:10.1002/cem.2555.
The ridge model fitting lmridge
, ridge CV and GCV plots cv.plot
, ridge AIC and BIC plots info.plot
, m-scale and isrm plots isrm.plot
, ridge and VIF trace plot.lmridge
, miscellaneous ridge plots rplots.plot
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.3, 0.002)) ## for indication vertical line (biasing parameter k) and ## horizontal line (minimum minimum ridge MSE values corresponding to vertical line) bias.plot(mod) ## without Horizontal and vertical line as set \code{abline = FALSE} bias.plot(mod, abline=FALSE)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.3, 0.002)) ## for indication vertical line (biasing parameter k) and ## horizontal line (minimum minimum ridge MSE values corresponding to vertical line) bias.plot(mod) ## without Horizontal and vertical line as set \code{abline = FALSE} bias.plot(mod, abline=FALSE)
Plot of ridge CV and GCV against scalar or vector values of biasing parameter (see Golub et al., 1979 <doi:10.1080/00401706.1979.10489751>).
cv.plot(x, abline = TRUE, ...)
cv.plot(x, abline = TRUE, ...)
x |
An object of class "lmridge". |
abline |
Horizontal and vertical lines to show minimum value of ridge GCV and CV at certain value of biasing parameter |
... |
Not presently used in this implementation. |
Function cv.plot
can be used to plot the values of ridge CV and GCV against scalar or vector value of biasing parameter . The
cv.plot
can be helpful for selection of optimal value of ridge biasing parameter . If no argument is used then horizontal line will indicate minimum GCV and Cv at certain value of biasing parameter
.
Nothing returned
Muhammad Imdad Ullah, Muhammad Aslam
Delaney, N. J. and Chatterjee, S. (1986). Use of the Bootstrap and Cross-Validation in Ridge Regression. Journal of Business & Economic Statistics. 4(2), 255–262.
Golub, G., Wahba, G. and Heat, C. (1979). Generalized Cross Validation as a Method for Choosing a Good Ridge Parameter. Technometrics. 21, 215–223. doi:10.2307/1268518.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
, bias variance trade-off plot bias.plot
, ridge AIC and BIC plots info.plot
, m-scale and isrm plots isrm.plot
, ridge and VIF trace plot.lmridge
, miscellaneous ridge plots rplots.plot
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.002)) ## for indication vertical line (biasing parameter k) and ## horizontal line (minimum respective CV and GCV values corresponding to vertical line) cv.plot(mod) ## without Horizontal and vertical line set \code{abline = FALSE} cv.plot(mod, abline = FALSE)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.002)) ## for indication vertical line (biasing parameter k) and ## horizontal line (minimum respective CV and GCV values corresponding to vertical line) cv.plot(mod) ## without Horizontal and vertical line set \code{abline = FALSE} cv.plot(mod, abline = FALSE)
Heat evolved during setting of 13 cement mixtures of four basic ingredients. Each ingredient percentage appears to be rounded down to a full integer. The sum of the four mixture percentages varies from a maximum of 99% to a minimum of 95%. If all four regressor X-variables always summed to 100%, the centered X-matrix would then be of rank only 3. Thus, the regression of heat on four X-percentages is ill-conditioned, with an approximate rank deficiency of MCAL = 1.
data(Hald)
data(Hald)
A data frame with 13 observations on the following 5 variables.
X1
p3ca: Integer percentage of 3CaO.Al2O3 in the mixture.
X2
p3cs: Integer percentage of 3CaO.SiO2 in the mixture.
X3
p4caf: Integer percentage of 4CaO.Al2O3.Fe2O3 in the mixture.
X4
p2cs: Integer percentage of 2CaO.SiO2 in the mixture.
y
hear: Heat (cals/gm) evolved in setting, recorded to nearest tenth.
The (lmridge) Hald data are identical to the (MASS) cement data except for variable names.
Woods, H., Steinour, H.H. and Starke, H.R. (1932). Effect of Composition of Portland Cement on Heat Evolved During Hardening. Industrial Engineering and Chemistry 24: 1207–1214.
Hald, A. (1952). Statistical Theory with Engineering Applications.(page 647.) New York; Wiley.
The hatr
function computes hat matrix (see Hastie and Tibshirani, 1990).
hatr(x, ...) ## S3 method for class 'lmridge' hatr(x, ...)
hatr(x, ...) ## S3 method for class 'lmridge' hatr(x, ...)
x |
An object of class "lmridge". |
... |
Not presently used in this implementation. |
Hat matrix for scalar or vector values of biasing parameter provided as argument to lmridge
. It is used to compute degrees of freedom for given , and error degree of freedom etc. The hat matrix can be computed using formula
equivalently
.
returns a list of matrix for each biasing parameter :
hatr |
A list of hat matrix for each biasing parameter |
.
Muhammad Imdad Ullah, Muhammad Aslam
Cule, E. and De lorio, M. (2012). A semi-Automatic method to guide the choice of ridge parameter in ridge regression. arXiv:abs/1205.0686v1 [stat.AP].
Hastie, T. and Tibshirani, R. (1990). Generalized Additive Models. Chapman and Hall.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
, ridge Var-Cov matrix vcov.lmridge
mod <- lmridge(y~., as.data.frame(Hald), K = c(0, 0.1, 0.2, 0.3)) ## Hat matrix for each biasing parameter hatr(mod) ## Hat matrix for first biasing parameter i.e. K = 0.1 hatr(mod)[[2]]
mod <- lmridge(y~., as.data.frame(Hald), K = c(0, 0.1, 0.2, 0.3)) ## Hat matrix for each biasing parameter hatr(mod) ## Hat matrix for first biasing parameter i.e. K = 0.1 hatr(mod)[[2]]
Plot of ridge AIC and BIC model selection criteria against ridge degrees of freedom (see Akaike, 1974 <doi:10.1109/TAC.1974.1100705>; Imdad, 2017 and Schwarz, 1978 <doi:10.1214/aos/1176344136>).
info.plot(x, abline = TRUE, ...)
info.plot(x, abline = TRUE, ...)
x |
An object of class "lmridge". |
abline |
Vertical line to show minimum value of ridge MSE at certain value of ridge degrees of freedom. |
... |
Not presently used in this implementation. |
Plot of ridge AIC and BIC against ridge degress of freedom . A vertical line represents the minimum ridge MSE at certain value of ridge df.
Nothing returned
Muhammad Imdad Ullah, Muhammad Aslam
Akaike, H. (1974). A new look at the Statistical Model Identification. IEEE Transaction on Automatic Control, 9(6), 716–723. doi:10.1109/TAC.1974.1100705.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Schwarz, G. (1978). Estimating the Dimension of a Model. Annals of Statistics, 6(2), 461–464. doi:10.1214/aos/1176344136.
The ridge model fitting lmridge
, ridge CV and GCV plotcv.plot
, variance biase trade-off plot bias.plot
, m-scale and isrm plots isrm.plot
, ridge and VIF trace plot.lmridge
, miscellaneous ridge plots rplots.plot
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.15, 0.002)) ## for indication vertical line (df ridge) info.plot(mod) ## without vertical line set \code{abline = FALSE} info.plot(mod, abline = FALSE)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.15, 0.002)) ## for indication vertical line (df ridge) info.plot(mod) ## without vertical line set \code{abline = FALSE} info.plot(mod, abline = FALSE)
The infocr.lmridge
function computes model information selection criteria (AIC and BIC), see Akaike, 1974 <doi:10.1109/TAC.1974.1100705>; Imdad, 2017 and Schwarz, 1978 <doi:10.1214/aos/1176344136>.
infocr(object, ...) ## S3 method for class 'lmridge' infocr(object, ...)
infocr(object, ...) ## S3 method for class 'lmridge' infocr(object, ...)
object |
An object of class "lmridge". |
... |
Not presently used in this implementation. |
Model information selection criteria are common way of selecting among model while balancing the competing goals of fit and parsimony. The model selection criteria AIC and BIC are computed by quantifying df
in the ridge regression model, using formula (). It can be helpful for selecting optimal value of biasing parameter
.
It returns a matrix of information criteria, AIC and BIC for each biasing parameter . Column of matrix indicates model selection criteria AIC and BIC, respectively, while rows indicate value of biasing parameter
for which model selection criteria are computed.
Muhammad Imdad Ullah, Muhammad Aslam
Akaike, H. (1974). A new look at the Statistical Model Identification. IEEE Transaction on Automatic Control, 9(6), 716-723. doi:10.1109/TAC.1974.1100705.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Schwarz, G. (1978). Estimating the Dimension of a Model. Annals of Statistics, 6(2), 461–464. doi:10.1214/aos/1176344136.
the ridge model fitting lmridge
, ridge AIC and BIC plot info.plot
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, .2, 0.001)) infocr(mod) ## Vector of AIC values infocr(mod)[,1] ## vector of BIC values infocr(mod)[,2]
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, .2, 0.001)) infocr(mod) ## Vector of AIC values infocr(mod)[,1] ## vector of BIC values infocr(mod)[,2]
Plot of m-scale and ISRM against scalar or vector values of biasing parameter (Vinod, 1976 <doi:10.1080/01621459.1976.10480955>).
isrm.plot(x, ...)
isrm.plot(x, ...)
x |
An object of class "lmridge". |
... |
Not presently used in this implementation. |
The isrm.plot
function can be used to plot the values of m-scale and ISRM against given list (scalar or vector values) of biasing parameter as argument to
lmridge
. It can be helpful for the optimal selection of the biasing parameter .
Nothing returned
Muhammad Imdad Ullah, Muhammad Aslam
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Vinod, H. (1976). Application of New Ridge Regression Methods to a Study of Bell System Scale Economics. Journal of the American Statistical Association, 71, 835–841. doi:10.2307/2286847.
The ridge model fitting lmridge
, ridge CV and GCV plots cv.plot
, ridge AIC and BIC plots info.plot
, variance bias trade-off plot bias.plot
, ridge and VIF trace plot.lmridge
, miscellaneous ridge plotsrplots.plot
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.002)) isrm.plot(mod) isrm.plot(mod, abline=FALSE)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.002)) isrm.plot(mod) isrm.plot(mod, abline=FALSE)
The kest
function computes different biasing parameters available in the literature proposed by different researchers.
kest(object, ...) ## S3 method for class 'lmridge' kest(object, ...) ## S3 method for class 'klmridge' print(x, ...)
kest(object, ...) ## S3 method for class 'lmridge' kest(object, ...) ## S3 method for class 'klmridge' print(x, ...)
object |
An object of class "lmridge" for the |
x |
An object of class "klmridge" for the |
... |
Not presently used in this implementation. |
The kest
function computes different biasing parameter for the ordinary linear ridge regression. All these methods are already available in the literature and proposed by various authors. See reference section.
The function returns the list of following biasing parameter methods proposed by various researchers.
mHKB |
By Thisted (1976), |
LW |
As in |
LW76 |
By Lawless and Wang (1976), |
CV |
Value of Cross Validation (CV) for each biasing parameter |
kCV |
Value of biasing parameter at which CV is small. |
HKB |
By Hoerl and Kennard (1970), |
kibAM |
By Kibria (2003), |
GCV |
Value of Generalized Cross Validation (GCV) for each biasing parameter |
kcGCV |
Value of biasing parameter at which GCV is small. |
DSK |
By Dwividi and Shrivastava, (1978), |
kibGM |
By Kibria (2003), |
kibMEd |
By Kibria (2003), |
KM2 |
By Muniz and Kibria (2009), |
KM3 |
By Muniz and Kibria (2009), |
KM4 |
By Muniz and Kibria (2009), |
KM5 |
By Muniz and Kibria (2009), |
KM6 |
By Muniz and Kibria (2009), |
KM8 |
By Muniz et al. (2012), |
KM9 |
By Muniz et al. (2012), |
KM10 |
By Muniz et al. (2012), |
KM11 |
By Muniz et al. (2012), |
KM12 |
By Muniz et al., |
KD |
By Dorugade and Kashid (2012), |
KAD4 |
By Dorugade and Kashid (2012), |
alphahat |
The OLS estimator in canonical form, i.e., |
Muhammad Imdad Ullah, Muhammad Aslam
Dorugade, A. and Kashid, D. (2010). Alternative Method for Choosing Ridge Parameter for Regression. Applied Mathematical Sciences, 4(9), 447-456.
Dorugade, A. (2014). New Ridge Parameters for Ridge Regression. Journal of the Association of Arab Universities for Basic and Applied Sciences, 15, 94-99. doi:10.1016/j.jaubas.2013.03.005.
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Kibria, B. (2003). Performance of Some New Ridge Regression Estimators. Communications in Statistics-Simulation and Computation, 32(2), 491-435. doi:10.1081/SAC-120017499.
Lawless, J., and Wang, P. (1976). A Simulation Study of Ridge and Other Regression Estimators. Communications in Statistics-Theory and Methods, 5(4), 307-323. doi:10.1080/03610927608827353.
Muniz, G., and Kibria, B. (2009). On Some Ridge Regression Estimators: An Empirical Comparisons. Communications in Statistics-Simulation and Computation, 38(3), 621-630. doi:10.1080/03610910802592838.
Muniz, G., Kibria, B., Mansson, K., and Shukur, G. (2012). On developing Ridge Regression Parameters: A Graphical Investigation. SORT-Statistics and Operations Research Transactions, 36(2), 115–138.
Thisted, R. A. (1976). Ridge Regression, Minimax Estimation and Empirical Bayes Methods. Technical Report 28, Division of Biostatistics, Stanford University, California.
Venables, W. N. and Ripley, B. D. (2002). Modern Applied Statistics with S. Springer New York, 4th edition, ISBN 0-387-95457-0.
The ridge model fitting lmridge
, Ridge Var-Cov matrix vcov
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.001)) kest(mod) ## GCV values kest(mod)$GCV ## minimum GCV value at certain k kest(mod)$kGCV ## CV Values kest(mod)$CV ## minimum CV value at certain k kest(mod)$kCV ## Hoerl and Kennard (1970) kest(mod)$HKB
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.001)) kest(mod) ## GCV values kest(mod)$GCV ## minimum GCV value at certain k kest(mod)$kGCV ## CV Values kest(mod)$CV ## minimum CV value at certain k kest(mod)$kCV ## Hoerl and Kennard (1970) kest(mod)$HKB
Fits a linear ridge regression model after scaling regressors and returns an object of class "lmridge" (by calling lmridgeEst
function) designed to be used in plotting method, testing of ridge coefficients and for computation of different ridge related statistics. The ridge biasing parameter can be a scalar or a vector. See Hoerl et al., 1975 <doi:10.1080/03610927508827232>, Horel and Kennard, 1970 <doi:10.1080/00401706.1970.10488634>.
lmridge(formula, data, K = 0, scaling=c("sc", "scaled", "non", "centered"), ...) lmridgeEst(formula, data, K=0, scaling=c("sc", "scaled", "non", "centered"), ...) ## Default S3 method: lmridge(formula, data, K = 0, scaling=c("sc", "scaled", "non", "centered"), ...) ## S3 method for class 'lmridge' coef(object, ...) ## S3 method for class 'lmridge' print(x, digits = max(5,getOption("digits") - 5), ...) ## S3 method for class 'lmridge' fitted(object, ...)
lmridge(formula, data, K = 0, scaling=c("sc", "scaled", "non", "centered"), ...) lmridgeEst(formula, data, K=0, scaling=c("sc", "scaled", "non", "centered"), ...) ## Default S3 method: lmridge(formula, data, K = 0, scaling=c("sc", "scaled", "non", "centered"), ...) ## S3 method for class 'lmridge' coef(object, ...) ## S3 method for class 'lmridge' print(x, digits = max(5,getOption("digits") - 5), ...) ## S3 method for class 'lmridge' fitted(object, ...)
formula |
Standard R formula expression, that is, a symbolic representation of the model to be fitted and has form |
data |
An optional data frame containing the variables in the model. If not found in data, the variables are taken from |
K |
Ridge biasing parameter (may be a vector). |
scaling |
The method to be used to scale the predictors. The scaling option |
object |
A lmridge object, typically generated by a call to |
x |
An object of class |
digits |
Minimum number of significant digits to be used. |
... |
Additional arguments to be passed to or from other methods. |
lmridge
or lmridgeEst
function fits in linear ridge regression after scaling the regressors and centering the response. The lmridge
is default a function that calls lmridgeEst
for computation of ridge coefficients and returns an object of class "lmridge" designed to be used in plotting method, testing of ridge coefficients and for computation of different ridge related statistics. If intercept is present in the model, its coefficient is not penalized. However, intercept is estimated from the relation .
print.lmridge
tries to be smart about formatting of ridge coefficients.
lmridge
function returns an object of class "lmridge" after calling list of named objects from lmridgeEst
function:
coef |
A named vector of fitted coefficients. |
call |
The matched call. |
Inter |
Was an intercept included? |
scaling |
The scaling method used. |
mf |
Actual data used. |
y |
The response variable. |
xs |
The scaled matrix of predictors. |
xm |
The vector of means of the predictors. |
terms |
The |
xscale |
Square root of sum of squared deviation from mean regarding the scaling option used in |
rfit |
The fitted value of ridge regression for given biasing parameter |
K |
The ridge regression biasing parameter |
d |
A vector of singular values of scaled |
div |
Eigenvalues of scaled regressors. |
Z |
A list of matrix |
The function at the current form cannot handle missing values. The user has to take prior action with missing values before using this function.
Muhammad Imdad Ullah, Muhammad Aslam
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.2307/1267351.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Testing of ridge coefficient summary.lmridge
data(Hald) mod <- lmridge(y~., data = as.data.frame(Hald), K = seq(0, 0.1, 0.01), scaling = "sc") ## Scaled Coefficients mod$coef ## Re-Scaled Coefficients coef(mod) ## ridge predicted values predict(mod) ## ridge residuals residuals(mod) ##ridge and VIF trace plot(mod) ## ridge VIF values vif(mod) ## ridge Var-Cov matrix vcov(mod) ## ridge biasing parameter by researchers kest(mod) ## ridge fitted values fitted(mod) ## ridge statistics 1 rstats1(mod) ## ridge statistics 2 rstats2(mod) ## list of objects from lmridgeEst function lmridgeEst(y~., data = as.data.frame(Hald), K = seq(0, 0.1, 0.01), scaling = "sc") lmridgeEst(y~., data = as.data.frame(Hald), K = seq(0, 0.1, 0.01), scaling = "non")
data(Hald) mod <- lmridge(y~., data = as.data.frame(Hald), K = seq(0, 0.1, 0.01), scaling = "sc") ## Scaled Coefficients mod$coef ## Re-Scaled Coefficients coef(mod) ## ridge predicted values predict(mod) ## ridge residuals residuals(mod) ##ridge and VIF trace plot(mod) ## ridge VIF values vif(mod) ## ridge Var-Cov matrix vcov(mod) ## ridge biasing parameter by researchers kest(mod) ## ridge fitted values fitted(mod) ## ridge statistics 1 rstats1(mod) ## ridge statistics 2 rstats2(mod) ## list of objects from lmridgeEst function lmridgeEst(y~., data = as.data.frame(Hald), K = seq(0, 0.1, 0.01), scaling = "sc") lmridgeEst(y~., data = as.data.frame(Hald), K = seq(0, 0.1, 0.01), scaling = "non")
Plot of VIF values (VIF trace) and ridge coefficients (ridge trace) for scalar or vector values of biasing parameter .
## S3 method for class 'lmridge' plot(x, type = c("ridge", "vif"), abline = TRUE, ...)
## S3 method for class 'lmridge' plot(x, type = c("ridge", "vif"), abline = TRUE, ...)
x |
An object of class "lmridge". |
type |
Either VIF trace or ridge trace. |
abline |
Horizontal and vertical line to show minimum value of MSE and GCV value at certain value of biasing parameter |
... |
Not presently used in this implementation. |
Graphical way of selecting optimal value of biasing parameter . The biasing parameter is selected when coefficients becomes stable in case of ridge trace. In cae of VIF trace
(ridge biasing parameter) can be selected for which VIF of each regressor near to one or value of
at which GCV is minimum. If no argument is used then all traces of ridge coefficients will be displayed. A vertical and horizontal line will also be displayed on ridge trace graph to indicate minimum ridge MSE (among the all computed ridge MSE based on provided vector of
) along with the value of respective biasing parameter
. For VIF trace, vetical line shows minmum GCV value at certain value of biasing parameter
.
Nothing
Muhammad Imdad Ullah, Muhammad Aslam
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
, ridge CV and GCV plots cv.plot
, variance bias trade-off plot bias.plot
, m-scale and isrm plots isrm.plot
, ridge AIC and BIC plots info.plot
, miscellaneous ridge plots rplots.plot
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.15, 0.002)) ## Ridge trace plot(mod) plot(mod, type = "ridge") ## VIF trace plot(mod, type = "vif") ## Ridge trace without abline plot(mod, type = "ridge", abline = FALSE)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.15, 0.002)) ## Ridge trace plot(mod) plot(mod, type = "ridge") ## VIF trace plot(mod, type = "vif") ## Ridge trace without abline plot(mod, type = "ridge", abline = FALSE)
Predicted values based on linear ridge regression model for scalar or vector values of biasing parameter .
## S3 method for class 'lmridge' predict(object, newdata, na.action=na.pass, ...)
## S3 method for class 'lmridge' predict(object, newdata, na.action=na.pass, ...)
object |
An object of class "lmridge". |
newdata |
An optional data frame in which to look for variables with which to predict. |
na.action |
Function determine what should be done with missing values in |
... |
Not presently used in this implementation. |
The predict.lmridge
function produces predicted values, obtained by evaluating the regression function in the frame newdata
which defaults to model.frame (object
). If newdata
is omitted the predictions are based on the data used for the fit. In that case how cases with missing values in the original fit are handled is determined by the na.action
argument of that fit. If na.action = na.omit
omitted cases will not appear in the predictions, whereas if na.action = na.exclude
they will appear (in predictions), with value NA.
predict.lmridge
produces a vector of predictions or a matrix of predictions for scalar or vector values of biasing parameter.
Variables are first looked for in newdata
and then are searched for in the usual way (which will include the environment of the formula used in the fit). A warning will be given if the variables found are not of the same length as those in the newdata
if it was supplied.
Muhammad Imdad Ullah, Muhammad Aslam
Cule, E. and De lorio, M. (2012). A semi-Automatic method to guide the choice of ridge parameter in ridge regression. arXiv:1205.0686v1 [stat.AP].
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
, ridge residuals residuals
, ridge PRESS press.lmridge
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.05)) predict(mod) predict(mod, newdata = as.data.frame(Hald[1:5, -1]))
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.05)) predict(mod) predict(mod, newdata = as.data.frame(Hald[1:5, -1]))
The press.lmridge
function computes predicted residual sum of squares (PRESS) (see Allen, 1971).
press(object, ...) ## S3 method for class 'lmridge' press(object, ...)
press(object, ...) ## S3 method for class 'lmridge' press(object, ...)
object |
An object of class "lmridge". |
... |
Not presently used in this implementation. |
All of the n
leave-one-out predicted residual sum of squares is calculated by fitting full regression model by using, , where
is hat matrix from ridge model fit,
is the ith residual at specific value of
.
The press.lmridge
produces a vector of PRESS or a matrix of PRESS for scalar or vector values of biasing parameter.
Muhammad Imdad Ullah, Muhammad Aslam
Allen, D. M. (1971). Mean Square Error of Prediction as a Criterion for Selecting Variables. Technometrics, 13, 469-475. doi:10.1080/00401706.1971.10488811.
Allen, D. M. (1974). The Relationship between Variable Selection and Data Augmentation and Method for Prediction. Technometrics, 16, 125-127. doi:10.1080/00401706.1974.10489157.
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
, ridge residual residuals
, ridge predicted value predict
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.5, 0.04)) press(mod)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.5, 0.04)) press(mod)
The residuals
function computes the ridge residuals for scalar or vector value of biasing parameter .
## S3 method for class 'lmridge' residuals(object, ...)
## S3 method for class 'lmridge' residuals(object, ...)
object |
An object of class "lmridge". |
... |
Not presently used in this implementation. |
The generic functions residuals
can be used to compute residuals object of linear ridge regression from lmridge
function.
Returns a vector or a matrix of ridge residuals for scalar or vector value biasing parameter provided as argument to
lmridge
function.
Muhammad Imdad Ullah, Muhammad Aslam
Berk, R. (2008). Statistical Learning from a Regression Perspective. Springer.
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Lee, W. F. (1979). Model Estimation Using Ridge Regression with the Variance Normalization Criterion. Master thesis, Department of Educational Foundation, Memorial University of Newfoundland.
The ridge mode fitting lmridge
, ridge prediction predict
, ridge PRESS values press
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 1, 0.2)) residuals(mod)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 1, 0.2)) residuals(mod)
Panel of three ridge related plots, df trace vs , RSS vs
and PRESS vs
for graphical judgement of optimal value of
.
rplots.plot(x, abline = TRUE, ...)
rplots.plot(x, abline = TRUE, ...)
x |
An object of class "lmridge" |
abline |
Vertical line to show minimum value of ridge PRESS at cartain value of biasing parameter |
... |
Not presently used in this implementation. |
Function rplots.plot
can be used to plot the values of df vs , RSS vs
and PRESS vs
for scalar or vector values of biasing parameter
. If no argument is used then a vertical line will be drawn on ridge PRESS plot to show the minimum value of PRESS at certain
. The panel of these three plots can be helful in selecting the optimal value of biasing parameter
.
nothing
Muhammad Imdad Ullah, Muhammad Aslam
Allen, D. M. (1971). Mean Square Error of Prediction as a Criterion for Selecting Variables. Technometrics, 13, 469-475. doi:10.1080/00401706.1971.10488811.
Allen, D. M. (1974). The Relationship between Variable Selection and Data Augmentation and Method for Prediction. Technometrics, 16, 125-127. doi:10.1080/00401706.1974.10489157.
Berk, R. (2008). Statistical Learning from a Regression Perspective. Springer.
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
, ridge CV and GCV plots cv.plot
, variance bias trade-off plot bias.plot
, m-scale and isrm plots isrm.plot
, ridge AIC and BIC plots info.plot
, ridge and VIF trace plot.lmridge
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.005)) rplots.plot(mod) rplots.plot(mod, abline = FALSE)
mod <- lmridge(y~., as.data.frame(Hald), K = seq(0, 0.2, 0.005)) rplots.plot(mod) rplots.plot(mod, abline = FALSE)
The rstats1
function computes the ordinary ridge related statistics such as variance, squared bias, MSE, R-squared and condition number (CN), etc. (see Lee, 1979; Kalivas and Palmer, 2014 <doi:10.1002/cem.2555>)
rstats1(x, ...) ## S3 method for class 'lmridge' rstats1(x, ...) ## S3 method for class 'rstats1' print(x, digits = max(5,getOption("digits") - 5), ...)
rstats1(x, ...) ## S3 method for class 'lmridge' rstats1(x, ...) ## S3 method for class 'rstats1' print(x, digits = max(5,getOption("digits") - 5), ...)
x |
An object of class "lmridge" (for the |
digits |
Minimum number of significant digits to be used for most numbers. |
... |
Not presently used in this implementation. |
The rstats1
function computes the ordinary ridge regression related statistics which may help in selecting optimal value of biasing parameter . If value of
is zero then these statistics are equivalent to the relevant OLS statistics.
Following are the ridge related statistics computed for given scalar or vector value of biasing parameter provided as argument to
lmridge
or lmridgeEst
function.
var |
Variance of ridge regression for given biasing parameter |
bias2 |
Squared bias of ridge regression for given biasing parameter |
mse |
Total MSE value for given biasing parameter |
Fv |
F-statistics value for testing of the significance of the ordinary ridge regression estimator computed for given biasing parameter |
rfact |
Shrinkage factor |
R2 |
R-squared for given biasing parameter |
adjR2 |
Adjusted R-squared for given biasing parameter |
eigval |
Eigenvalue of |
CN |
Condition number after addition of biasing parameter in |
Muhammad Imdad Ullah, Muhammad Aslam
Cule, E. and De lorio, M. (2012). A semi-Automatic method to guide the choice of ridge parameter in ridge regression. arXiv:1205.0686v1 [stat.AP].
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Kalivas, J. H., and Palmer, J. (2014). Characterizing multivariate calibration tradeoffs (bias, variance, selectivity, and sensitivity) to select model tuning parameters. Journal of Chemometrics, 28(5), 347–357. doi:10.1002/cem.2555.
Ridge related statistics rstats2
, the ridge model fitting lmridge
, ridge var-cov matrix vcov
data(Hald) mod <- lmridge(y~., data = as.data.frame(Hald), K = seq(0,0.2, 0.005) ) rstats1(mod) ## Getting only Ridge MSE rstats1(mod)[3] rstats1(mod)$mse
data(Hald) mod <- lmridge(y~., data = as.data.frame(Hald), K = seq(0,0.2, 0.005) ) rstats1(mod) ## Getting only Ridge MSE rstats1(mod)[3] rstats1(mod)$mse
The rstats2
function computes the ordinary ridge related statistics such as ,
, ridge degrees of freedom, effective degrees of freedom (EDF), and prediction residual error sum of squares PRESS statistics for scalar or vector value of biasing parameter
(See Allen, 1974 <doi:10.2307/1267500>; Lee, 1979; Hoerl and Kennard, 1970 <doi:10.2307/1267351>).
rstats2(x, ...) ## S3 method for class 'lmridge' rstats2(x, ...) ## S3 method for class 'rstats2' print(x, digits = max(5,getOption("digits") - 5), ...)
rstats2(x, ...) ## S3 method for class 'lmridge' rstats2(x, ...) ## S3 method for class 'rstats2' print(x, digits = max(5,getOption("digits") - 5), ...)
x |
For the |
digits |
Minimum number of significant digits to be used. |
... |
Not presently used in this implementation. |
The rstats2
function computes the ridge regression related different statistics which may help in selecting the optimal value of biasing parameter . If value of
is zero then these statistics are equivalent to the relevant OLS statistics.
Following are ridge related statistics computed for given scalar or vector value of biasing parameter provided as argument to
lmridge
or lmridgeEst
function.
CK |
|
dfridge |
DF of ridge for given biasing parameter |
EP |
Effective number of Parameters for given biasing parameter |
redf |
Residual effective degrees of freedom for given biasing parameter |
EF |
Effectiveness index for given biasing parameter |
ISRM |
Quantification of concept of stable region proposed by Vinod and Ullah, 1981, i.e., |
m |
m-scale for given value of biasing parameter proposed by Vinod (1976) alternative to plotting of the ridge coefficients, i.e., |
PRESS |
PRESS statistics for ridge regression introduced by Allen, 1971, 1974, i.e., |
Muhammad Imdad Ullah, Muhammad Aslam
Allen, D. M. (1971). Mean Square Error of Prediction as a Criterion for Selecting Variables. Technometrics, 13, 469-475. doi:10.1080/00401706.1971.10488811.
Allen, D. M. (1974). The Relationship between Variable Selection and Data Augmentation and Method for Prediction. Technometrics, 16, 125-127. doi:10.1080/00401706.1974.10489157.
Cule, E. and De lorio, M. (2012). A semi-Automatic method to guide the choice of ridge parameter in ridge regression. arXiv:1205.0686v1 [stat.AP].
Hastie, T. and Tibshirani, R. (1990). Generalized Additive Models. Chapman & Hall.
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Kalivas, J. H., and Palmer, J. (2014). Characterizing Multivariate Calibration Tradeoffs (Bias, Variance, Selectivity, and Sensitivity) to Select Model Tuning Parameters. Journal of Chemometrics, 28(5), 347–357. doi:10.1002/cem.2555.
Lee, W. F. (1979). Model Estimation Using Ridge Regression with the Variane Normalization Criterion. Master thesis, Department of Educational Foundation Memorial University of Newfoundland.
Ridge related statistics rstats1
, ridge model fitting lmridge
data(Hald) mod <- lmridge(y~., data=as.data.frame(Hald), K = seq(0,0.2, 0.001) ) rstats2(mod)
data(Hald) mod <- lmridge(y~., data=as.data.frame(Hald), K = seq(0,0.2, 0.001) ) rstats2(mod)
The summary
method for class "lmridge" for scalar or vector biasing parameter (Cule and De lorio, 2012).
## S3 method for class 'lmridge' summary(object, ...) ## S3 method for class 'summary.lmridge' print(x, digits = max(3, getOption("digits") - 3), signif.stars = getOption("show.signif.stars"), ...)
## S3 method for class 'lmridge' summary(object, ...) ## S3 method for class 'summary.lmridge' print(x, digits = max(3, getOption("digits") - 3), signif.stars = getOption("show.signif.stars"), ...)
object |
An "lmridge" object, typically generated by a call to |
x |
An object of class |
signif.stars |
logical: if |
digits |
The number of significant digits to use when printing. |
... |
Not presently used in this implementation. |
print.summary.lmridge
tries to be smart about formatting the coefficients, standard errors etc. and additionally gives 'significance stars' if signif.stars
is TRUE
.
The function summary
computes and returns a list of summary statistics of the fitted linear ridge regression model for scalar or vector value biasing parameter given as argument in
lmridge
function. All summary information can be called using list object summaries
.
coefficients |
A |
stats |
Ridge related statistics of R-squared, adjusted R-squared, F-statistics for testing of coefficients, AIC and BIC values for given biasing parameter |
rmse1 |
Minimum MSE value for given biasing parameter |
rmse2 |
Value of |
K |
Value of given biasing parameter. |
df1 |
Numerator degrees of freedom for p-value of F-statistics. |
df2 |
Denominator degrees of freedom for p-value of F-statistics. |
fpvalue |
p-value for each F-statistics. |
Muhammad Imdad Ullah, Muhammad Aslam
Cule, E. and De lorio, M. (2012). A semi-Automatic method to guide the choice of ridge parameter in ridge regression. arXiv:1205.0686v1 [stat.AP].
Hoerl, A. E., Kennard, R. W., and Baldwin, K. F. (1975). Ridge Regression: Some Simulation. Communication in Statistics, 4, 105-123. doi:10.1080/03610927508827232.
Hoerl, A. E. and Kennard, R. W., (1970). Ridge Regression: Biased Estimation of Nonorthogonal Problems. Technometrics, 12, 55-67. doi:10.1080/00401706.1970.10488634.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
The ridge model fitting lmridge
mod <- lmridge(y~., as.data.frame(Hald), K = c(0, 0.0132, 0.1)) summary(mod) ## coefficients for first biasing parameter summary(mod)$summaries[[1]]$coefficients summary(mod)$summaries[[1]][[1]] ## ridge related statistics from summary function summary(mod)$summaries[[1]]$stats ## Ridge F-test's p-value summary(mod)$summaries[[1]]$fpvalue
mod <- lmridge(y~., as.data.frame(Hald), K = c(0, 0.0132, 0.1)) summary(mod) ## coefficients for first biasing parameter summary(mod)$summaries[[1]]$coefficients summary(mod)$summaries[[1]][[1]] ## ridge related statistics from summary function summary(mod)$summaries[[1]]$stats ## Ridge F-test's p-value summary(mod)$summaries[[1]]$fpvalue
The vcov
function computes the variance-covariance matrix for the estimates of linear ridge regression model.
## S3 method for class 'lmridge' vcov(object, ...)
## S3 method for class 'lmridge' vcov(object, ...)
object |
For |
... |
Not presently used in this implementation. |
The vcov
function computes variance-covariance matrix for scalar or vector value of biasing parameter provided as argument to
lmridge
function.
A list of matrix of estimated covariances in the linear ridge regression model for scalar or vector biasing parameter K
K is produced. Each list element has row and column names corresponding to the parameter names given by the coef(mod)
. List items are named correspond to values of biasing parameter .
Covariance will be without intercept term, as intercept term is not penalized in ridge regression.
Muhammad Imdad Ullah, Muhammad Aslam
Brown, G.W. and Beattie, B.R. (1975). Improving Estimates of Economic Parameters by use of Ridge Regression with Production Function Applications. American Journal of Agricultural Economics, 57(1), 21-32. doi:10.2307/1238836.
The ridge model fitting lmridge
, ridge VIF values vif
data(Hald) mod<- lmridge(y~., data=as.data.frame(Hald), scaling="sc", K=seq(0,1,.2) ) vcov.lmridge(mod) vcov(mod)
data(Hald) mod<- lmridge(y~., data=as.data.frame(Hald), scaling="sc", K=seq(0,1,.2) ) vcov.lmridge(mod) vcov(mod)
Computes VIF values for each scalar or vector value of biasing parameter (Marquardt, 1970).
vif(x, ...) ## S3 method for class 'lmridge' vif(x, ...)
vif(x, ...) ## S3 method for class 'lmridge' vif(x, ...)
x |
For VIF method, an object of class "lmridge", i.e., a fitted model. |
... |
Not presently used in this implementation. |
The vif.lmridge
function computes VIF value for each regressor in data set after addition of biasing parameter as argument to lmridge
function. The VIF is computed using , given by Marquardt, (1970).
The vif
function returns a matrix of VIF values for each regressor after adding scalar or vector biasing parameter to
matrix. The column of returned matrix indicates regressors name and row indicates value of each biasing parameter
provided as argument to
lmridge
function.
Muhammad Imdad Ullah, Muhammad Aslam
Fox, J. and Monette, G. (1992). Generalized Collinearity Diagnostics. JASA, 87, 178–183.
Imdad, M. U. Addressing Linear Regression Models with Correlated Regressors: Some Package Development in R (Doctoral Thesis, Department of Statistics, Bahauddin Zakariya University, Multan, Pakistan), 2017.
Marquardt, D. (1970). Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation. Technometrics, 12(3), 591–612.
The ridge model fitting lmridge
, ridge Var-Cov matrix vcov
data(Hald) mod <- lmridge(y~., data = as.data.frame(Hald), scaling = "sc", K = seq(0,1,.2) ) vif(mod)
data(Hald) mod <- lmridge(y~., data = as.data.frame(Hald), scaling = "sc", K = seq(0,1,.2) ) vif(mod)