DMBoot(data ; alpha, bootinput_kwargs...)

Method type for doing a Diebold-Mariano test using a dependent bootstrap. A constructor that takes the data and several keyword arguments is provided. Relevant keyword arguments are:

alpha=0.05 <- The confidence level to use in the test 

blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

See the DependentBootstrap package docs for more info on bootstrap input keyword arguments.

DMHAC(data ; alpha::Float64=0.05, kernelfunction, bandwidth::Int=-1)

Method type for doing a Diebold-Mariano test using a HAC variance estimator. A constructor that takes the data and three keyword arguments is provided. Relevant keyword arguments are:

alpha <- The confidence level for the test 

kernelfunction <- Kernel function to use in HAC variance estimator. Valid values are
    KernelEpanechnikov(), KernelGaussian(), KernelUniform(), KernelBartlett(). You can
    also :epanechnikov, :gaussian, :uniform, :bartlett, or String variants of these symbols. 

bandwidth <- Bandwidth to use in HAC variance estimator (set less than or equal to -1 to estimate bandwidth using Politis (2003) "Adaptive Bandwidth Choice")

DMMethod

Abstract type for nesting the various methods that can be used to perform Diebold-Mariano tests. Subtypes are:

DMHAC
DMBoot

The subtypes have entries in the help (?) menu.

DMTest(rejH0::Int, pvalue::Float64, bestinput::Int, teststat::Float64, dmmethod::DMMethod)

Output type for a Diebold-Mariano test. A description of the fields follows:

rejH0 <- true if the null is rejected, false otherwise
pvalue <- p-value from the test
bestinput <- 1 if forecast 1 is more accurate, and 2 if forecast 2 is more accurate. See ?dm for definition of forecast 1 versus 2.
teststat <- If dmmethod == :hac then is the mean of loss difference scaled by HAC variance
            If dmmethod == :boot then is the mean of loss difference
dmmethod <- Diebold-Mariano method used in the test. See ?DMMethod for more detail.

MCSBoot(data ; alpha, basecaseindex, kwargs...)

Method type for doing a MCS test using a dependent bootstrap. Relevant keyword arguments are:

alpha=0.05 <- The confidence level to use in the test 

basecaseindex=1 <- The MCS does not have a natural basecase. However, the input data is a
	matrix of losses, not loss differentials, and the method itself considers every possible
	loss differential combination. This is far too many to check for determining
	the optimal block length for bootstrapping, so users who wish to auto-estimate
	the appropriate block length can specify a notional base case variable index
	for the purposes of computing a optimal block lengths of all loss differences
	relative to the base case.
blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

Note, see the DependentBootstrap docs for more information on valid keyword arguments for the BootInput constructor.

MCSBootLowRAM(data ; alpha::Float64=0.05, kwargs...)

Method type for doing an MCS test using a dependent bootstrap. This method type has an identical constructor arguments to MCSBoot, so use ?MCSBoot for more detail.

WARNING: This method corresponds to a MCS algorithm that has double the runtime of MCSBoot, but uses about half as much RAM. The vast majority of users will want to use MCSBoot.

WARNING: Results from MCSBootLowRAM are not guaranteed to be identical to those from MCSBoot in finite sample, although they are very likely to both recommend the same set of models in the model confidence set.

MCSMethod

Abstract type for nesting the various methods that can be used to perform a model confidence set. Subtypes are:

MCSBoot
MCSBootLowRAM

The subtypes have entries in the help (?) menu.

MCSTest(inQF::Vector{Int}, outQF::Vector{Int}, pvalueQF::Vector{Float64}, inMT::Vector{Int}, outMT::Vector{Int}, pvalueMT::Vector{Float64})

Output type from performing the MCS test proposed in Hansen, Lunde, Nason (2011) "The Model Confidence Set", Econometrica, 79 (2), pp. 453-497.

The field names of this type have trailing "QF" or "MT", where QF corresponds to the quadratic form test (see section 3.1.1 of original paper), while MT corresponds to the maximum t-stat test (see section 3.1.2 of original paper).

Note, in the fields of this type, the forecast models input to the MCS method are indicated via an integer counting up from 1 for each forecast model. These integers correspond to the columns of the input loss matrix; see ?mcs for more info.

The fields of this type follow:

inQF <- Models that are in the MCS via the quadratic form method
outQF <- Models that are not in the MCS via the quadratic form method
pvalueQF <- The cumulative p-values from the quadratic form method
inMT <- Models that are in the MCS via the max t-stat method
outMT <- Models that are not in the MCS via the max t-stat method
pvalueMT <- The cumulative p-values from the max t-stat method

RCBoot(data ; alpha::Float64=0.05, bootinput_kwargs...)

Method type for doing a Reality Check using a dependent bootstrap. A constructor that takes the data and several keyword arguments is provided. The data refers to the loss differences for each forecast relative to the basecase. Use ?rc for more detail. Relevant keyword arguments are: alpha=0.05 <- The confidence level to use in the test

kernelfunction=KernelEpanechnikov() <- The kernel function to use with HAC variance estimator. See ?hacvariance for more detail. 

bandwidth=-1 <- The bandwidth for HAC variance estimator. If bandwidth <= -1 then bandwidth is estimated using Politis (2003) "Adaptive Bandwidth Choice" 

blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

See the DependentBootstrap package docs for more info on bootstrap input keyword arguments.

RCMethod

Abstract type for nesting the various methods that can be used to perform a Reality Check. Subtypes are:

RCBoot

The subtypes have entries in the help (?) menu.

RCTest(rejH0::Int, pvalue::Float64)

Output type from a Reality Check test. The fields of this type follow:

rejH0 <- true if the null is rejected, false otherwise
pvalue <- p-value from the test

SPABoot(data ; alpha, kernelfunction, bandwidth, bootinput_kwargs...)

Method type for doing an SPA test using a dependent bootstrap. A constructor that takes the data and several keyword arguments is provided. The data refers to the loss differences for each forecast relative to the basecase. Use ?spa for more detail. Relevant keyword arguments are:

alpha=0.05 <- The confidence level to use in the test 

kernelfunction=KernelEpanechnikov() <- The kernel function to use with HAC variance estimator. See ?hacvariance for more detail. 

bandwidth=-1 <- The bandwidth for HAC variance estimator. If bandwidth <= -1 then bandwidth is estimated using Politis (2003) "Adaptive Bandwidth Choice" 

blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

Note, see the DependentBootstrap docs for more information on valid keyword arguments for the BootInput constructor.

SPAMethod

Abstract type for nesting the various methods that can be used to perform an SPA test. Subtypes are:

SPABoot

The subtypes have entries in the help (?) menu.

SPATest(rejH0::Int, mu_u::Vector{Float64}, mu_c::Vector{Float64}, mu_l::Vector{Float64}, pvalue_u::Float64, pvalue_c::Float64, pvalue_l::Float64, pvalue::Float64, teststat::Float64)

Output type for an SPA test proposed in Hansen (2005) "A Test for Superior Predictive Ability". The names of the fields for this type follow those in the referenced paper, so see that paper for more detail. The fields of this type follow:

rejH0 <- true if the null is rejected, false otherwise
mu_u <- mu associated with upper bound on threshold rate
mu_c <- Recommended mu
mu_l <- mu associated with lower bound on threshold rate
pvalue_u <- pvalue associated with mu_u
pvalue_c <- pvalue assoicated with mu_c
pvalue_l <- pvalue associated with mu_l
pvalueauto <- Recommended p-value (usually pvalue_c except in rare situations where this method can fail)
teststat::Float64 <- SPA test statistic (look for tSPA in source paper)

Note the different mu variables, {u, c, l} are described in the referenced article on p370-371.

Most users will just be interested in the field pvalueauto.

dm(lossdiff::Vector{<:Number}, dmmethod ; kwargs...)

This function implements the test proposed in Diebold, Mariano (1995) "Comparing Predictive Accuracy".

Let x1 denote forecast 1, x2 denote forecast 2, and let y denote the forecast target. Let L(., .) denote a loss function. Then the first argument lossdiff is assumed to be a vector created by the following operation:

L(x1, y) - L(x2, y)

The second argument, dmmethod, can be an explicit method type, currently DMHAC or DMBoot, (see ?DMHAC and ?DMBoot for more detail), in which case the keyword arguments are not needed.

Alternatively, dmmethod can be set to the Symbol :hac or :boot, depending on whether the user wants to use the HAC method or the bootstrap method. In this instance, the keyword arguments provided will be passed on to DMHAC or DMBoot constructors (see ?DMHAC and ?DMBoot for more detail).

Finally, the second argument can be omitted entirely, in which case the method will default to the default bootstrap method, which is the stationary bootstrap of Politis and Romano (1993) with block length estimated followed Patton, Politis, and White (2009).

The output of a Diebold-Mariano test is of type DMTest. Use ?DMTest for more information.

hacvariance{T<:Number}(x::AbstractVector{T} ; kf::Symbol=:epanechnikov, bw::Int=-1)::Tuple{Float64, Int}

Get the heteroskedasticity and autocorrelation consistent variance estimator of data vector x. The function has the following keyword arguments: kf <- Kernel function used in estimator. Valid values are :epanechnikov, :gaussian, :uniform, :bartlett bw <- Bandwidth used in estimator. If <= -1, then estimate bandwidth using Politis (2003) "Adaptive Bandwidth Choice"

lossdiffbase_case <- Local function for getting loss differences relative to notional base case for purposes of estimating the block length

mcs(l::Matrix{<:Number}, method ; kwargs...)
mcs(l::Matrix{<:Number} ; kwargs...)

This function implements the MCS test proposed in Hansen, Lunde, Nason (2011) "The Model Confidence Set", Econometrica, 79 (2), pp. 453-497.

Let x_k, k = 1, ..., K, denote K forecasts (from K different forecasting models), and y denote the forecast target. Let L(., .) denote a loss function. The first argument of mcs is a matrix where the kth column of the matrix is created by the operation:

L(x_k, y)

Note that unlike the Reality Check and SPA test, there is no base case for the MCS.

The second method argument determines which methodology to use. Currently, only MCSBoot is available and if this input type is provided, the keyword arguments are not needed. Alternatively, the user can omit the second argument, and then any keyword arguments will be passed to the MCSBoot constructor. See ?MCSBoot for more detail. The most relevant keyword arguments are: alpha=0.05 <- The confidence level to use in the test

basecaseindex=1 <- The MCS does not have a natural basecase. However, the input data is a
	matrix of losses, not loss differentials, and the method itself considers every possible
	loss differential combination. This is far too many to check for determining
	the optimal block length for bootstrapping, so users who wish to auto-estimate
	the appropriate block length can specify a notional base case variable index
	for the purposes of computing a optimal block lengths of all loss differences
	relative to the base case.
blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

For more detail on the bootstrap options, please see the docs for the DependentBootstrap package.

The output of a MCS test is of type MCSTest. Use ?MCSTest for more information.

Note, if you are hitting RAM limits, type ?ForecastEval.MCSBootLowRAM at the REPL for more detail on an alternative algorithm that is also available.

Note, for any developers, the main algorithm (associated with MCSBoot) still has the following potential performance issues:

ISSUE 1: Some of the temporary arrays in the loops could probably be eliminated 

ISSUE 2: For MCS method A, I think the loop over K could be terminated as soon as cumulative p-values are greater than method.alpha. Need to double check this. 

ISSUE 3: Need to add option to do just max(abs) method or just sum(sq) method (or both)

Comments or pull requests on these issues would be most welcome on the package github page.

pvaluelocal(d::ContinuousUnivariateDistribution, x::Number ; tail::Symbol=:both)::Float64
pvaluelocal{T<:Number}(d::Vector{T}, x::Number ; tail::Symbol=:both, as::Bool=false)::Float64

Obtain the p-value associated with the inputs. d::ContinuousUnivariateDistribution -> d is the distribution under the null and x is the test statistic. d::Vector{T} -> d is a bootstrapped vector of test statistics and x is the value of the test statistic under the null.

rc(lD::Matrix{T}, method ; kwargs)
rc(lD::Matrix{T} ; kwargs)

This function implements the test proposed in White (2000) "A Reality Check for Data Snooping" following the methodology in Hansen (2005).

Let x0 denote a base-case forecast, xk, k = 1, ..., K, denote K alternative forecasts, and y denote the forecast target. Let L(., .) denote a loss function. The first argument of rc is a matrix where the kth column of the matrix is created by the operation:

L(xk, y) - L(x0, y)

Note that the forecast loss comes first and the base case loss comes second. This is the opposite to what is described in White's paper.

The second method argument determines which methodology to use. Currently, only RCBoot is available and if this input type is provided, the keyword arguments are not needed. Alternatively, the user can omit the second argument, and then any keyword arguments will be passed to the RCBoot constructor. See ?RCBoot for more detail. The most relevant keyword arguments are: alpha=0.05 <- The confidence level to use in the test

blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

The output of a Reality Check test is of type RCTest. Use ?RCTest for more information.

spa{T<:Number}(lossDiff::Matrix{T}, method ; kwargs...)::SPATest
spa{T<:Number}(lossDiff::Matrix{T} ; kwargs...)

This function implements the SPA test proposed in Hansen (2005) "A Test for Superior Predictive Ability".

Let x0 denote a base-case forecast, xk, k = 1, ..., K, denote K alternative forecasts, and y denote the forecast target. Let L(., .) denote a loss function. The first argument of spa is a matrix where the kth column of the matrix is created by the operation:

L(xk, y) - L(x0, y)

Note that the forecast loss comes first and the base case loss comes second. This is the opposite to what is described in Hansen's paper.

The second method argument determines which methodology to use. Currently, only SPABoot is available and if this input type is provided, the keyword arguments are not needed. Alternatively, the user can omit the second argument, and then any keyword arguments will be passed to the SPABoot constructor. See ?SPABoot for more detail. The most relevant keyword arguments are: alpha=0.05 <- The confidence level to use in the test

kernelfunction=KernelEpanechnikov() <- The kernel function to use with HAC variance estimator. See ?hacvariance for more detail. 

bandwidth=-1 <- The bandwidth for HAC variance estimator. If bandwidth <= -1 then bandwidth is estimated using Politis (2003) "Adaptive Bandwidth Choice" 

blocklength=0.0 <- Block length to use with the bootstrap. Default of 0.0 implies
    the block length will be optimally estimated from the data use the method deemed
    most appropriate by the DependentBootstrap package (typically the selection procedure
    of Patton, Politis, and White (2009)). 

numresample=1000 <- Number of resamples to use when bootstrapping. 

bootmethod=:stationary <- Bootstrap methodology to use, where the default is the
    stationary bootstrap of Politis and Romano (1993)

The output of an SPA test is of type SPATest. Use ?SPATest for more information.

Note, Hansen suggests using the Stationary Bootstrap implied HAC variance estimator, which is not currently supported in this package. However, note that any consistent HAC estimator is valid and in many cases may be preferred.