subsampling LOO estimates with diff-est-srs-wor start #496
Conversation
This reverts commit 781e331.
|
I wanted to use R2, and as I had rewrote summary stats anyway, added R2 and made all mse, rmse, and R2 to use only normal approximation with as much shared computation as possible With the added R2 support, this PR will close also #483 |
|
I can take a look |
|
I have added some comments, but I'm not done with the review yet. Besides, I think documentation needs to be updated (at least re-roxygenized, but also progressr should be mentioned), the vignettes perhaps as well, and I haven't run |
new column behavior of `summary.vsel()`, and the omittance of option `"best"` of argument `baseline`
|
Commit a7458b9 reverts the changes with respect to the new "mixed deltas" variant of Now I still need to do points 1 and 3 from #496 (comment) and, as mentioned in #496 (comment), some other things are still missing (documentation, vignettes, |
Branch |
It is not mentioned there, but it is required to get the appropriate variance for the plots. I first followed the paper, and then the variance can be really small, and then thinking it through realized that at least in our use case we need to add them together. |
|
btw. I had kept |
Makes sense. Did I break something with respect to |
It's not used, but I thought it's better to keep it there until the incremental sub-sampling is supported, and then decide whether to keep depending on how the incremental sub-sampling is implemented. |
subsampled LOO (`n_loo < n_full`) and everything else (`n_loo == n_full`)
does not change from definition of `var_mse_e` to use of `var_mse_e`, the extra definition of `var_mse_e` can be avoided
`is.null(summaries_baseline)`); one reason is that I'm not sure whether it was supposed to read `mu_baseline <- y` in that case (instead of `mu_baseline <- 0`)
| # simple transformation of mse | ||
| value <- sqrt(mse_e) - ifelse(is.null(summaries_baseline), 0, sqrt(mse_b)) | ||
| # the first-order Taylor approximation of the variance | ||
| value_se <- sqrt(value_se^2 / mse_e / 4) |
There was a problem hiding this comment.
If is.null(summaries_baseline) (i.e., deltas = FALSE), this line should be correct. But if !is.null(summaries_baseline) (i.e., deltas = TRUE), isn't mse_b also a random variable that needs to be taken into account? That would require a bivariate (instead of scalar) delta method (which is not a problem, because the delta method has a straightforward multivariate extension). What I mean is that the function for which the Taylor series approximation is made would then be
| value.se <- weighted.sd((mu - y)^2 - (mu.bs - y)^2, wcv, | ||
| na.rm = TRUE) / | ||
| sqrt(n_notna) | ||
| # Use normal approximation for mse and delta method for rmse and R2 |
There was a problem hiding this comment.
Is this supposed to refer to the standard error estimation method or the CI method? The first part ("normal approximation") refers to a CI method, but the second part ("delta method") to a standard error estimation method.
| # Compute mean and variance in log scale by matching the variance of a | ||
| # log-normal approximation |
There was a problem hiding this comment.
Is it common to assume a log-normal distribution as the sampling distribution of the MSE and RMSE estimators? I haven't seen that yet (I think), but it might be perfectly fine (motivated by the central limit theorem, I guess).
| # store for later calculations | ||
| mse_e <- value | ||
| if (!is.null(summaries_baseline)) { | ||
| # delta=TRUE, variance of difference of two normally distributed |
There was a problem hiding this comment.
There is something missing at the end; perhaps "random variables"?
| ((mu_baseline - y)^2 - mse_b))[loo_inds], | ||
| y_idx = loo_inds, | ||
| w = wobs) | ||
| cov_mse_e_b <- srs_diffe$y_hat / n_full^2 |
There was a problem hiding this comment.
Just for my understanding: This procedure for estimating the covariance between mse_e and mse_b assumes that the summands within mse_e and mse_b coming from different observations are uncorrelated, right? At first, I thought this was violated here because mu, mu_baseline, and summaries_fast$mu are model-based and hence there is potential for cross-observations dependencies, but then I realized that mu, mu_baseline, and summaries_fast$mu all are based on the leave-one-out principle, so is this the reason why we can assume a cross-observation correlation of zero here?
|
When I changed several bootstraps to analytic approximations and improved other approximations, I thought the math I was writing in the code was so trivial that I didn't write all the derivations and assumptions separately. Now I see I should have done that, as it takes also me a bit of time to re-check any of these when you ask a question, so they are not as obvious as I thought them to be. If you like, I can some day write the equations and assumptions for easier checking. Before that, at least every approximation I wrote is based on the tests at least as accurate as the earlier bootstrap, but much faster. |
| mse_y <- mean(wobs * (mean(y) - y)^2) | ||
| value <- 1 - mse_e / mse_y - ifelse(is.null(summaries_baseline), 0, 1 - mse_b / mse_y) | ||
| # the first-order Taylor approximation of the variance | ||
| var_mse_y <- .weighted_sd((mean(y) - y)^2, wobs)^2 / n_full |
There was a problem hiding this comment.
For mean(y), don't we need to take wobs into account? I think this is similar to line var_mse_b <- .weighted_sd((mu_baseline - y)^2, wobs)^2 / n_full where the parameter estimates from which mu_baseline is computed also take wobs into account.
There was a problem hiding this comment.
This also concerns several other occurrences of mean(y) here in get_stat().
| if (!is.null(summaries_baseline)) { | ||
| # delta=TRUE | ||
| mse_e <- mse_e - mse_b | ||
| } | ||
| value_se <- sqrt((value_se^2 - | ||
| 2 * mse_e / mse_y * cov_mse_e_y + | ||
| (mse_e / mse_y)^2 * var_mse_y) / mse_y^2) |
There was a problem hiding this comment.
If my understanding from stan-dev/loo#205 (comment) is correct, then I think we would need a trivariate delta method in the !is.null(summaries_baseline) case (because mse_b comes in, too). I haven't checked whether such a trivariate delta method would give the same formula as used here. Have you checked this?
For me, does not work (runs into some error, similarly as works for me. Does it work for you as well (on |
|
With |
That sounds good. The warnings probably arise from the first creation of the snapshots. If you are running the tests via Since this solution seems to be working for you, I will push a commit to |
…of `cv_varsel()`
This fixes commit 0d73c8e. However, before commit 0d73c8e, `is.null(mu_baseline)` should have never occurred because if `summaries_baseline` was `NULL`, then `mu_baseline` was set to `0` (and if `summaries_baseline` was not `NULL`, then `mu_baseline` was set to `summaries_baseline$mu` which should not be `NULL` either). Hence, this fixup here does not only fix commit 0d73c8e, but also the incorrect behavior which existed before it.
| # log-normal approximation | ||
| # https://en.wikipedia.org/wiki/Log-normal_distribution#Arithmetic_moments | ||
| mul <- log(value^2 / sqrt(value_se^2 + value^2)) | ||
| varl <- log(1 + value_se^2 / value^2) |
There was a problem hiding this comment.
Would it make sense to use log1p() here (for numerical stability)?
Work in progress.
Tested with
Notes
cv_varselif nloo<n and fast PSIS-LOO result is not yet available, fast PSIS-LOO result is computedcv_varselif nloo<n, fast PSIS-LOO result is stored in slot $summaries_fastNext
tagging @n-kall