Imputing outliers in tibble after identifying them
up vote
0
down vote
favorite
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
add a comment |
up vote
0
down vote
favorite
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
r dataframe outliers tbl
asked Nov 21 at 22:38
Nina
52
52
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421412%2fimputing-outliers-in-tibble-after-identifying-them%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown