subset using percentile for gridded data











up vote
0
down vote

favorite












I have gridded data that has 24249 obs and 963 var for daily maximum temperatures (K). I am looking for a way in r to select all days with maximum temperatures higher than the 90th percentile.



> dim(DailyT)

[1] 24249   963

> DailyT[1:4,1:7]

     x    y  1988-05-01 1988-05-02 1988-05-03 1988-05-04 1988-05-05

1 34.000 33   291.7603   291.8044   291.6158   292.9659   293.7032

2 34.125 33   291.7240   291.7951   291.5439   292.9451   293.7017

3 34.250 33   291.6884   291.7866   291.4721   292.9250   293.7001

4 34.375 33   291.6521   291.7781   291.4010   292.9049   293.6986


I did this but did not work 



df<- DailyT[DailyT[,3:963] <= quantile(DailyT[,3:963],.9, na.rm = T, type = 6) ]





r







share|improve this question






















  • Maybe you find this helpful.
    – A. Suliman
    Nov 22 at 9:07















up vote
0
down vote

favorite












I have gridded data that has 24249 obs and 963 var for daily maximum temperatures (K). I am looking for a way in r to select all days with maximum temperatures higher than the 90th percentile.



> dim(DailyT)

[1] 24249   963

> DailyT[1:4,1:7]

     x    y  1988-05-01 1988-05-02 1988-05-03 1988-05-04 1988-05-05

1 34.000 33   291.7603   291.8044   291.6158   292.9659   293.7032

2 34.125 33   291.7240   291.7951   291.5439   292.9451   293.7017

3 34.250 33   291.6884   291.7866   291.4721   292.9250   293.7001

4 34.375 33   291.6521   291.7781   291.4010   292.9049   293.6986


I did this but did not work 



df<- DailyT[DailyT[,3:963] <= quantile(DailyT[,3:963],.9, na.rm = T, type = 6) ]





r







share|improve this question






















  • Maybe you find this helpful.
    – A. Suliman
    Nov 22 at 9:07













up vote
0
down vote

favorite









up vote
0
down vote

favorite











I have gridded data that has 24249 obs and 963 var for daily maximum temperatures (K). I am looking for a way in r to select all days with maximum temperatures higher than the 90th percentile.



> dim(DailyT)

[1] 24249   963

> DailyT[1:4,1:7]

     x    y  1988-05-01 1988-05-02 1988-05-03 1988-05-04 1988-05-05

1 34.000 33   291.7603   291.8044   291.6158   292.9659   293.7032

2 34.125 33   291.7240   291.7951   291.5439   292.9451   293.7017

3 34.250 33   291.6884   291.7866   291.4721   292.9250   293.7001

4 34.375 33   291.6521   291.7781   291.4010   292.9049   293.6986


I did this but did not work 



df<- DailyT[DailyT[,3:963] <= quantile(DailyT[,3:963],.9, na.rm = T, type = 6) ]





r







share|improve this question













I have gridded data that has 24249 obs and 963 var for daily maximum temperatures (K). I am looking for a way in r to select all days with maximum temperatures higher than the 90th percentile.



> dim(DailyT)

[1] 24249   963

> DailyT[1:4,1:7]

     x    y  1988-05-01 1988-05-02 1988-05-03 1988-05-04 1988-05-05

1 34.000 33   291.7603   291.8044   291.6158   292.9659   293.7032

2 34.125 33   291.7240   291.7951   291.5439   292.9451   293.7017

3 34.250 33   291.6884   291.7866   291.4721   292.9250   293.7001

4 34.375 33   291.6521   291.7781   291.4010   292.9049   293.6986


I did this but did not work 



df<- DailyT[DailyT[,3:963] <= quantile(DailyT[,3:963],.9, na.rm = T, type = 6) ]





r




r






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 22 at 8:55









Ali

337




337












  • Maybe you find this helpful.
    – A. Suliman
    Nov 22 at 9:07


















  • Maybe you find this helpful.
    – A. Suliman
    Nov 22 at 9:07
















Maybe you find this helpful.
– A. Suliman
Nov 22 at 9:07




Maybe you find this helpful.
– A. Suliman
Nov 22 at 9:07












1 Answer
1






active

oldest

votes

















up vote
0
down vote













First, you need an id column to identify the rows later. Then, calculate the 90% quantile of all temperature values. At the end subset data witch any row cells exceeding q.



DailyT <- cbind(id=rownames(DailyT), DailyT)  # to identify rows later

q <- quantile(as.matrix(DailyT[, -(1:3)]), .9, na.rm = T, type = 6)  # 293.7003

DailyT.q <- DailyT[which(sapply(1:nrow(DailyT), function(x) any(DailyT[x, -(1:2)] >= q))), ]


Yields



> DailyT.q

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017


Edit:
To get the quantile rowwise use apply()



q90 <- apply(DailyT[, 4:8], MARGIN=1, quantile, .9,na.rm = T, type = 6)



> data.frame(DailyT, q90=q90)

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05      q90

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032 293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017 293.7017

3  3 34.250 33    291.6884    291.7866    291.4721    292.9250    293.7001 293.7001

4  4 34.375 33    291.6521    291.7781    291.4010    292.9049    293.6986 293.6986


Data



> dput(DailyT)

structure(list(x = c(34, 34.125, 34.25, 34.375), y = c(33L, 33L, 

                                                       33L, 33L), X1988.05.01 = c(291.7603, 291.724, 291.6884, 291.6521

                                                       ), X1988.05.02 = c(291.8044, 291.7951, 291.7866, 291.7781), X1988.05.03 = c(291.6158, 

                                                                                                                                   291.5439, 291.4721, 291.401), X1988.05.04 = c(292.9659, 292.9451, 

                                                                                                                                                                                 292.925, 292.9049), X1988.05.05 = c(293.7032, 293.7017, 293.7001, 

                                                                                                                                                                                                                     293.6986)), class = "data.frame", row.names = c(NA, -4L))





share|improve this answer























  • Thanks, I need to calculate the 90% quantile of each row not of all data.
    – Ali
    Nov 24 at 11:09










  • Aha, please see my edit.
    – jay.sf
    Nov 24 at 11:24










  • Worked.... Many thanks
    – Ali
    Nov 24 at 13:32










  • Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
    – jay.sf
    Nov 24 at 13:58













Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53427098%2fsubset-using-percentile-for-gridded-data%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote













First, you need an id column to identify the rows later. Then, calculate the 90% quantile of all temperature values. At the end subset data witch any row cells exceeding q.



DailyT <- cbind(id=rownames(DailyT), DailyT)  # to identify rows later

q <- quantile(as.matrix(DailyT[, -(1:3)]), .9, na.rm = T, type = 6)  # 293.7003

DailyT.q <- DailyT[which(sapply(1:nrow(DailyT), function(x) any(DailyT[x, -(1:2)] >= q))), ]


Yields



> DailyT.q

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017


Edit:
To get the quantile rowwise use apply()



q90 <- apply(DailyT[, 4:8], MARGIN=1, quantile, .9,na.rm = T, type = 6)



> data.frame(DailyT, q90=q90)

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05      q90

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032 293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017 293.7017

3  3 34.250 33    291.6884    291.7866    291.4721    292.9250    293.7001 293.7001

4  4 34.375 33    291.6521    291.7781    291.4010    292.9049    293.6986 293.6986


Data



> dput(DailyT)

structure(list(x = c(34, 34.125, 34.25, 34.375), y = c(33L, 33L, 

                                                       33L, 33L), X1988.05.01 = c(291.7603, 291.724, 291.6884, 291.6521

                                                       ), X1988.05.02 = c(291.8044, 291.7951, 291.7866, 291.7781), X1988.05.03 = c(291.6158, 

                                                                                                                                   291.5439, 291.4721, 291.401), X1988.05.04 = c(292.9659, 292.9451, 

                                                                                                                                                                                 292.925, 292.9049), X1988.05.05 = c(293.7032, 293.7017, 293.7001, 

                                                                                                                                                                                                                     293.6986)), class = "data.frame", row.names = c(NA, -4L))





share|improve this answer























  • Thanks, I need to calculate the 90% quantile of each row not of all data.
    – Ali
    Nov 24 at 11:09










  • Aha, please see my edit.
    – jay.sf
    Nov 24 at 11:24










  • Worked.... Many thanks
    – Ali
    Nov 24 at 13:32










  • Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
    – jay.sf
    Nov 24 at 13:58

















up vote
0
down vote













First, you need an id column to identify the rows later. Then, calculate the 90% quantile of all temperature values. At the end subset data witch any row cells exceeding q.



DailyT <- cbind(id=rownames(DailyT), DailyT)  # to identify rows later

q <- quantile(as.matrix(DailyT[, -(1:3)]), .9, na.rm = T, type = 6)  # 293.7003

DailyT.q <- DailyT[which(sapply(1:nrow(DailyT), function(x) any(DailyT[x, -(1:2)] >= q))), ]


Yields



> DailyT.q

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017


Edit:
To get the quantile rowwise use apply()



q90 <- apply(DailyT[, 4:8], MARGIN=1, quantile, .9,na.rm = T, type = 6)



> data.frame(DailyT, q90=q90)

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05      q90

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032 293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017 293.7017

3  3 34.250 33    291.6884    291.7866    291.4721    292.9250    293.7001 293.7001

4  4 34.375 33    291.6521    291.7781    291.4010    292.9049    293.6986 293.6986


Data



> dput(DailyT)

structure(list(x = c(34, 34.125, 34.25, 34.375), y = c(33L, 33L, 

                                                       33L, 33L), X1988.05.01 = c(291.7603, 291.724, 291.6884, 291.6521

                                                       ), X1988.05.02 = c(291.8044, 291.7951, 291.7866, 291.7781), X1988.05.03 = c(291.6158, 

                                                                                                                                   291.5439, 291.4721, 291.401), X1988.05.04 = c(292.9659, 292.9451, 

                                                                                                                                                                                 292.925, 292.9049), X1988.05.05 = c(293.7032, 293.7017, 293.7001, 

                                                                                                                                                                                                                     293.6986)), class = "data.frame", row.names = c(NA, -4L))





share|improve this answer























  • Thanks, I need to calculate the 90% quantile of each row not of all data.
    – Ali
    Nov 24 at 11:09










  • Aha, please see my edit.
    – jay.sf
    Nov 24 at 11:24










  • Worked.... Many thanks
    – Ali
    Nov 24 at 13:32










  • Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
    – jay.sf
    Nov 24 at 13:58















up vote
0
down vote










up vote
0
down vote









First, you need an id column to identify the rows later. Then, calculate the 90% quantile of all temperature values. At the end subset data witch any row cells exceeding q.



DailyT <- cbind(id=rownames(DailyT), DailyT)  # to identify rows later

q <- quantile(as.matrix(DailyT[, -(1:3)]), .9, na.rm = T, type = 6)  # 293.7003

DailyT.q <- DailyT[which(sapply(1:nrow(DailyT), function(x) any(DailyT[x, -(1:2)] >= q))), ]


Yields



> DailyT.q

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017


Edit:
To get the quantile rowwise use apply()



q90 <- apply(DailyT[, 4:8], MARGIN=1, quantile, .9,na.rm = T, type = 6)



> data.frame(DailyT, q90=q90)

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05      q90

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032 293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017 293.7017

3  3 34.250 33    291.6884    291.7866    291.4721    292.9250    293.7001 293.7001

4  4 34.375 33    291.6521    291.7781    291.4010    292.9049    293.6986 293.6986


Data



> dput(DailyT)

structure(list(x = c(34, 34.125, 34.25, 34.375), y = c(33L, 33L, 

                                                       33L, 33L), X1988.05.01 = c(291.7603, 291.724, 291.6884, 291.6521

                                                       ), X1988.05.02 = c(291.8044, 291.7951, 291.7866, 291.7781), X1988.05.03 = c(291.6158, 

                                                                                                                                   291.5439, 291.4721, 291.401), X1988.05.04 = c(292.9659, 292.9451, 

                                                                                                                                                                                 292.925, 292.9049), X1988.05.05 = c(293.7032, 293.7017, 293.7001, 

                                                                                                                                                                                                                     293.6986)), class = "data.frame", row.names = c(NA, -4L))





share|improve this answer














First, you need an id column to identify the rows later. Then, calculate the 90% quantile of all temperature values. At the end subset data witch any row cells exceeding q.



DailyT <- cbind(id=rownames(DailyT), DailyT)  # to identify rows later

q <- quantile(as.matrix(DailyT[, -(1:3)]), .9, na.rm = T, type = 6)  # 293.7003

DailyT.q <- DailyT[which(sapply(1:nrow(DailyT), function(x) any(DailyT[x, -(1:2)] >= q))), ]


Yields



> DailyT.q

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017


Edit:
To get the quantile rowwise use apply()



q90 <- apply(DailyT[, 4:8], MARGIN=1, quantile, .9,na.rm = T, type = 6)



> data.frame(DailyT, q90=q90)

  id      x  y X1988.05.01 X1988.05.02 X1988.05.03 X1988.05.04 X1988.05.05      q90

1  1 34.000 33    291.7603    291.8044    291.6158    292.9659    293.7032 293.7032

2  2 34.125 33    291.7240    291.7951    291.5439    292.9451    293.7017 293.7017

3  3 34.250 33    291.6884    291.7866    291.4721    292.9250    293.7001 293.7001

4  4 34.375 33    291.6521    291.7781    291.4010    292.9049    293.6986 293.6986


Data



> dput(DailyT)

structure(list(x = c(34, 34.125, 34.25, 34.375), y = c(33L, 33L, 

                                                       33L, 33L), X1988.05.01 = c(291.7603, 291.724, 291.6884, 291.6521

                                                       ), X1988.05.02 = c(291.8044, 291.7951, 291.7866, 291.7781), X1988.05.03 = c(291.6158, 

                                                                                                                                   291.5439, 291.4721, 291.401), X1988.05.04 = c(292.9659, 292.9451, 

                                                                                                                                                                                 292.925, 292.9049), X1988.05.05 = c(293.7032, 293.7017, 293.7001, 

                                                                                                                                                                                                                     293.6986)), class = "data.frame", row.names = c(NA, -4L))






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 24 at 11:24

























answered Nov 22 at 9:21









jay.sf

4,21621436




4,21621436












  • Thanks, I need to calculate the 90% quantile of each row not of all data.
    – Ali
    Nov 24 at 11:09










  • Aha, please see my edit.
    – jay.sf
    Nov 24 at 11:24










  • Worked.... Many thanks
    – Ali
    Nov 24 at 13:32










  • Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
    – jay.sf
    Nov 24 at 13:58




















  • Thanks, I need to calculate the 90% quantile of each row not of all data.
    – Ali
    Nov 24 at 11:09










  • Aha, please see my edit.
    – jay.sf
    Nov 24 at 11:24










  • Worked.... Many thanks
    – Ali
    Nov 24 at 13:32










  • Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
    – jay.sf
    Nov 24 at 13:58


















Thanks, I need to calculate the 90% quantile of each row not of all data.
– Ali
Nov 24 at 11:09




Thanks, I need to calculate the 90% quantile of each row not of all data.
– Ali
Nov 24 at 11:09












Aha, please see my edit.
– jay.sf
Nov 24 at 11:24




Aha, please see my edit.
– jay.sf
Nov 24 at 11:24












Worked.... Many thanks
– Ali
Nov 24 at 13:32




Worked.... Many thanks
– Ali
Nov 24 at 13:32












Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
– jay.sf
Nov 24 at 13:58






Very good! - Please mark the question as answered when you're satisfied with the given answer and win +2 reputation. This stops people spending time on answering a question that has already been answered.
– jay.sf
Nov 24 at 13:58




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53427098%2fsubset-using-percentile-for-gridded-data%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Trompette piccolo

Image
Ne doit pas être confondu avec trompette de poche. Trompette piccolo en si bémol, avec des branches d'embouchure différentes pour le si bémol (la plus courte) ou la (la plus longue) Trompette piccolo La trompette piccolo est la plus petite de la famille des trompettes. Elle joue une octave plus haut que la trompette standard en si bémol. La plupart des trompettes piccolo sont conçues pour jouer dans les deux tonalités de si bémol ou la, en utilisant une branche d'embouchure différente pour chaque tonalité. Le tube de la trompette piccolo en si bémol a une longueur moitié de celle de la trompette normale en si bémol. Des trompettes piccolo en sol, fa, ut sont également fabriquées, mais elles sont rares. La trompette soprano en ré, aussi connu comme la « trompette Bach », a été inventée vers 1890 par le facteur belge Victor-Charles Mahillon pour jouer les parties hautes de la trompette dans la musique de Bach et Haendel. La trompette piccolo moderne permet aux
Read more

Slow SSRS Report in dynamic grouping and multiple parameters

Image
0 I have developed a report which has 5 dynamic row groups from where users can choose in parameter selection. The report has 12 parameters so that user can get required information as per their parameter choice. Each parameter has multi value selection option with numerous lists of values (example: Customer parameter has 35,000 values and Item parameter has 2,500 values). Now, the report takes huge time to load parameter values as well as slow in preview (although, server resource is adequate). I am looking for a solution to improve the parameter selection performance without reducing the number of parameter and dynamic grouping sql-server reporting-services ssrs-2012 share | improve this question
Read more

Simon Yates (cyclisme)

Image
Pour les articles homonymes, voir Simon Yates et Yates. Simon Yates.mw-parser-output .entete.cyclisme{background-image:url("//upload.wikimedia.org/wikipedia/commons/thumb/8/86/Cycling_%28road%29_pictogram.svg/45px-Cycling_%28road%29_pictogram.svg.png")} Simon Yates lors du Tour d'Alberta 2014 Informations Naissance 7 août 1992 (26 ans) Bury Nationalité Britannique Équipe actuelle Mitchelton-Scott Spécialités Grimpeur, cyclisme sur piste Équipes amateurs 2013 100% Me Équipes professionnelles 01.2014-06.2016 [ n 1 ] Orica-GreenEDGE 07.2016-12.2016 [ n 2 ] Orica-BikeExchange 2017 Orica-Scott 2018- Mitchelton-Scott Principales victoires 1 classement mondial UCI World Tour 2018 1 titre mondial Champion du monde de course aux points (2013) 1 grand tour Tour d'Espagne 2018 2 classements annexes de grands tours Classement du meilleur jeune Tour de France 2017 Classement du combiné Tour d'
Read more