# Discontinuities due to survey redesigns : a structural time series approach

an irregular term ɛ_{t}, where t = (1, . . . , t) is the observation period. In this paper for the victimization

series there are several observations at one time point, so the subscript k is used to refer to series k at

time point t.

In this paper the measurement error model as described in van den Brakel and Renssen (2005) is implemented.

They consider the observed population value obtained via complete enumeration under two or

more survey designs as the sum of the true population value and a bias induced by the survey design.

Hence, Y_{t,k }= Y ^{true}_{t,k }+ ∂_{t,k}, where Y_{t,k }is the population value observed under a survey at time t, Y ^{true}

t,k

is the true unobserved population value at time t and ∂_{t,k }is the measurement bias introduced by the

survey at time t to measure the true population value Y ^{true}_{t,k }. More specific, using the decomposition as

proposed by Durbin and Koopman (2001), let

Y ^{true}_{t,k }= µ_{t,k }+ ι_{t,k }+ γ_{t,k }+ ɛ_{t,k}, with t = (1, . . . , t) and k = (1, . . . , k),

where ɛ_{t,k }∼ N ^{(}0, σ^{2}^{)}

_{ɛ }and k refers to the number of series observed at a time point. If a survey is

redesigned at time τ the value for the new survey Y ^{∗ ∗}

τ,k

becomes Y_{τ,k}

true

= Y_{τ,k }+ ∂^{∗}_{τ,k}. Then, the difference

Y ^{∗}_{τ,k }− Y_{τ,k }= ∂^{∗}_{τ,k }− ∂_{τ,k }is the systematic difference between two survey approaches caused by the

introduction of a new survey as discussed in section 3.3.

As proposed by Durbin and Koopman (2001), it is possible to include a regression variable δ_{t }with its

time independent coefficient β_{k}, such that explanatory variables can be allowed in the structural model

framework. Following van den Brakel and Roels (2010), in this paper the regression variable δ_{t }is used as

an intervention variable to quantify discontinuities. The intervention coefficient β_{k }models the difference

∂^{∗}

t,k

− ∂_{t,k}.

Hence, the model boils down to

Y_{t,k }= µ_{t,k }+ ι_{t,k }+ γ_{t,k }+ β_{k}δ_{t }+ ɛ_{t,k}, with t = (1, . . . , t) and k = (1, . . . , k). (4.1)

Note that (4.1) does not account for any sampling error. Following Scott and Smith (1974), Y_{t,k }can

be considered as the true population value which can be properly described with the structural time

series model (4.1). Then, let ˆy^{(}^{greg}^{)}_{t,k }= Y_{t,k }+ e_{t,k}. Here, ˆy^{(}^{greg}^{)}_{t,k }is the parameter value estimated by

the generalized regression estimator as discussed in section 2.4, Y_{t,k }the unobserved population value

described by (4.1) and e_{t,k }is the sampling error since not the entire population but a sample is observed.

Summarized, the estimated values for the target parameters become

ˆy^{(}^{greg}^{)}_{t,k }= µ_{t,k }+ ι_{t,k }+ γ_{t,k }+ β_{k}δ_{t }+ ɛ_{t,k }+ e_{t,k}. (4.2)

20

To improve readability, the superscript in ˆy^{(}^{greg}^{)}_{t,k }will be left out in this section unless it is necessary.

Also, note that in this entire section the components containing the time index t should also contain the

label l to denote the survey type considered at time t. Since in the dataset considered there is only one

overlapping survey, the SM_{IV }, the time index t is sufficient to indicate the survey type. In case SM_{IV }is

used, it will be explicitly stated to to avoid ambiguities.

4.2.2 Application to victimization data

In this paper for the victimization series a relatively short time span of 19 years is considered. It is

therefore unlikely that the cyclic effects of (4.2) will have significant effects, and the same holds for the

seasonal effects since we have annual observations. Therefore, following the approach in van den Brakel

and Roels (2010), it has been decided to disregard the seasonal component ι_{t,k }and the cyclic component

γ_{t,k}. For the dataset considered in this paper, it is possible to check for this assumption by looking at

the Durbin-Watson statistics or the ACF plots.

In (4.2), the sampling errors e_{t,k }can be correlated through time. As proposed by Binder and Dick (1990),

using the variances of ˆy_{t,k }the following form can be obtained which allows for nonhomogeneous variance

in the sampling errors :

where

e_{t,k }=

√

V ar^{( )}

ˆy_{t,k }× ˜e_{t,k}, (4.3)

√

V ar^{( )}

ˆy_{t,k }is the standard error of ˆy_{t,k }defined in section 2.4, and ˜e_{t,k }is an ARMA process

which models the serial correlation between the sampling errors. Since the standard error of ˆy_{t,k }is not

available for all surveys considered in this paper, this approach cannot be implemented here. To account

for nonhomogeneity between the sampling errors it is assumed that the sampling errors are inversely

proportional to the sample size of the survey.

Since in this paper only cross sectional series are considered, it is not possible to separate e_{t,k }from ɛ_{t,k},

which would be possible if a panel dataset was available. Therefore, it has been decided to combine the

sampling error e_{tk }and the irregular term ɛ_{t,k }into one irregular term ϕ_{t,k }= e_{t,k }+ ɛ_{t,k}. The variance of

ϕ_{t,k }is inversely proportional to the sample size n_{t}, such that ϕ_{t,k }^{∼ }^{(}

_{= }N 0, ^{σ}^{2}

ϕ,k

n_{t}

)

. This also implies that

the sampling error e_{t,k }dominates the irregular term ɛ_{t,k }(van den Brakel and Roels, 2010). Note that

since no panel surveys are considered in this analysis, it can be assumed that Cov(ϕ_{t,k}, ϕ_{t}^{′}_{,k}) = 0 ∀ t ̸= t^{′}.

These simplifications result in the following linear Gaussian model for the structural time series analysis :

ˆy_{t,k }= µ_{t,k }+ β_{k}δ_{t }+ ϕ_{t,k}. (4.4)

Stochastic trend

The first term in equation (4.4), µ_{t,k}, is the component which models the stochastic trend in the model.

21

More specific, the smooth trend model is assumed, which is defined as

µ_{t,k }= µ_{t}_{−}_{1}_{,k }+ r_{t}_{−}_{1}_{,k}, where r_{t,k }= r_{t}_{−}_{1}_{,k }+ η_{t,r,k}.

Here µ_{t,k }is called the level component, r_{t,k }the stochastic slope component and η_{t,r,k }^{∼}

= ^{N}

(

0, σ^{2}

)

r,k

the irregular component of the trend. The restriction that the irregular terms of the stochastic trend

are independent, Cov(η_{t,r,k}, η_{t}^{′}_{,r,k}^{′}) = 0, t ̸= t^{′ }and k ̸= k^{′}, holds in the first part of the analysis, and

is released when explanatory variables are introduced. Durbin and Koopman (2001) also considers the

local linear trend model, which also includes an irregular term in µ_{t,k}. It has been decided to disregard

this irregular term because it is not expected that the population is changing rapidly.

Intervention variable

As aforementioned, the terms β_{k}δ_{t }are the core elements of this analysis, they model the systematic

difference between two different survey approaches, as discussed in section 3.3. To quantify the effects

of a survey redesign, the regression variable δ_{t }is used as an intervention variable with its regression

coefficient β_{k }(van den Brakel and Roels, 2010). A first time application of an intervention analysis is the

paper by Harvey and Durbin (1986) on the implication of the British seat belt law. Depending on the

type of intervention, δ_{t }can be defined in several ways. In case of a level intervention, it is assumed that

the magnitude of the discontinuity is constant over time after a survey redesign occurred at time τ :

⎧^{⎪⎨ }1 t ≥ τ

δ_{t }=

^{⎪⎩ }0 t < τ

. (4.5)

This approach models only one survey redesign, which is easily extended to more interventions as it is

the case in this paper :

δ^{i}

t ^{=}

⎧

⎪_{⎨}

⎪_{⎩}

1 τ_{i }≤ t < τ_{i}_{+1}

0 otherwise

, (4.6)

where i refers to the ith survey redesign, τ_{i }is the time point of that survey redesign and τ_{i}_{+1 }is the

time point of the next survey redesign. Other examples of intervention variables are the pulse and slope

intervention variables (Durbin and Koopman, 2001) :

⎧^{⎪⎨ }1 t = τ

δ_{t }=

^{⎪⎩ }0 t ̸= τ

⎧^{⎪⎨ }1 + t − τ t ≥ τ^{, δ}^{t }^{= }^{⎪⎩ }0 t < τ

.

The regression coefficient of the intervention variable has no irregular term. There are examples when

a survey redesign also leads to an increase or decrease in ϕ_{t,k }as well. Model (4.3) can be used to incorporate

the estimated variances of the survey estimates as prior information. Another possibility to

account for that would be introducing separate variances of the measurement equations every time a

22

new survey is introduced. More specific, if survey l1 has been redesigned at time τ to survey l2, then

V ar(ϕ_{t,k}) = σ_{ϕ,k,l}_{1 }if t < τ and V ar(ϕ_{t,k}) = σ_{ϕ,k,l}_{2 }if t ≥ τ. To test the equivalence of these variances,

a sufficient amount of observations per survey is needed. Due to the rather short time span of the victimization

series this is not possible, and as proposed by van den Brakel and Roels (2010) it has been

decided to disregard this effect.

There is one exceptional case which should be investigated more closely. Suppose a new survey is introduced

at time point τ. On top of that, suppose that at τ there is also a change in the trend of the

parameter of interest, say due to an economic crisis. If one would apply the intervention analysis, the

intervention coefficient will wrongly assign the effect of a crisis to the new survey. Therefore, a crucial assumption

is that there is no structural change in neither the stochastic trend nor in the possible seasonal

components, such that all discontinuities can be only due to the introduction of a new survey. To check

for this assumption, one could incorporate an explanatory variable which does contain any structural

changes and which trend is correlated with the trend of the parameter of interest. In this paper, this

explanatory variable is the police series as described in section 3.2.2.

4.2.3 Transition into a state space model

Up till now, the series as provided by Statistics Netherlands have been modeled in a structural

time series model. Note that the discontinuities have not been estimated yet, the components are only

classified. Using the Kalman filter the discontinuities can be estimated, as described in the next section. To

estimate the parameters of a structural time series model, it has to be put in a state space representation.

The transformation of model (4.4) to a state space model as it had been addressed by Durbin and

Koopman (2001) is described in this subsection. In detail, a linear Gaussian state space model represents

a set of k equations coupled in a time-varying system :

ˆy_{t }= c_{t }+ Z_{t}α_{t }+ ϕ_{t},

α_{t}_{+1 }= d_{t }+ T_{t}α_{t }+ η_{t}.

(4.7a)

(4.7b)

Variable Dimension Variable Dimension

ˆy_{t}, c_{t}, ϕ_{t }k × 1 Z_{t }k × m

α_{t}, d_{t}, η_{t }m × 1 T_{t }m × m

k =number of variables, m = dimension of the state variable.

Table 4 – Dimensions of the variables as described in the linear Gaussian state space form (4.7)

Here, k is the number of variables per time point t. For the victimization series, k = 6 except for the

subpopulations, where k = 5. The dimensions of the matrices and vectors are given in table 4.

Equation (4.7a) is called the measurement equation, and equation (4.7b) is referred to as the transition

23

equation. Note that the constants c_{t }and d_{t }are set equal to zero in this paper, but they can be used to

implement known constant effects of developments.

The measurement equation describes how the observed series ˆy_{t }depends on the underlying unobserved

state variable α_{t }via the matrix Z_{t}. The state variable α_{t }contains the parameters of the level and the

slope and the regression coefficients of the intervention variables discussed in section 4.2.2. The transition

equation describes how the state variable α_{t }develops over time via the matrix T_{t}. The matrices Z_{t }and

T_{t }are referred to as system matrices. In case any of the matrices is time invariant, the time subscript is

dropped.

To get from (4.4) to (4.7), the matrices get the following values :

α_{t }= (µ_{t,}_{1}, r_{t,}_{1}, . . . , µ_{t,}_{k}, r_{t,}_{k}, β_{1}, . . . , β_{k})^{′ },

(4.8a)

Z_{t }= (I_{k }⊗ (1, 0)|δ_{t}I_{k}), T = Blockdiag(T_{tr}, I_{k}), T_{tr }= I_{k }⊗ ( ^{1 1}

0 1

) , (4.8b)

ϕ_{t }= (ϕ_{t,}_{1}, . . . , ϕ_{t,}_{k})^{′}, η_{t }= (0, η_{t,r,}_{1}, . . . , 0, η_{t,r,}_{k}, 0^{′}_{k})^{′}, E(ϕ_{t}) = 0_{k}, E(η_{t}) = 0_{3}_{k},

H_{t }= Cov(ϕ_{t}) = ^{1 }Diag(σ^{2}_{ϕ,}_{1}, . . . , σ^{2}

ϕ,k^{)}^{,}

n_{t}

Q_{t }= Cov(η_{t}) = Diag(0, σ^{2}_{r,}_{1}, . . . , 0, σ^{2}

r,k^{, }^{0}^{′}

k^{)}^{.}

(4.8c)

(4.8d)

(4.8e)

Here, I_{k }is a k×k identity matrix, ⊗ denotes the Kronecker product, δ_{t }is defined in (4.5), Blockdiag(x, y)

creates a diagonal matrix with x and y on the diagonals and 0_{k }is a column vector of order k. The

variances of the disturbances are summarized in covariance matrices H_{t }and Q_{t}.

Note that (4.8) is only defined for one survey redesign. If a series has more than one redesign, the model

can be easily extended. In the case of the victimization series there are three survey redesigns, such that

α_{t }= ^{(}µ_{t,}_{1}, r_{t,}_{1}, . . . , µ_{t,}_{k}, r_{t,}_{k}, β^{D}^{1}_{1 }, . . . , β^{D}^{1}

k

, β^{D}^{2}

_{1 }, . . . , β^{D}^{2}

k

, β^{D}^{3}

_{1 }, . . . , β^{D}^{3}

k

Z_{t }= (I_{k }⊗ (1, 0)|(1, 1, 1) ⊗ δ^{i}_{t}I_{k}), T = Blockdiag(T_{tr}, I_{3}_{k}), T_{tr }= I_{k }⊗ ( ^{1 1}

0 1

)_{′}

, (4.9a)

) , (4.9b)

ϕ_{t }= (ϕ_{t,}_{1}, . . . , ϕ_{t,}_{k})^{′}, η_{t }= (0, η_{t,r,}_{1}, . . . , 0, η_{t,r,}_{k}, 0^{′}_{3}_{k})^{′}, E(ϕ_{t}) = 0_{k}, E(η_{t}) = 0_{5}_{k},

H_{t }= Cov(ϕ_{t}) = ^{1 }Diag(σ^{2}_{ϕ,}_{1}, . . . , σ^{2}

ϕ,k^{)}^{,}

n_{t}

Q_{t }= Cov(η_{t}) = Diag(0, σ^{2}_{r,}_{1}, . . . , 0, σ^{2}

r,k^{, }^{0}^{′}

3k^{)}^{.}

(4.9c)

(4.9d)

(4.9e)

Recall that δ^{i}_{t }is defined in (4.6) for the 3 interventions denoted by D1, D2 and D3, so there are 3 × k

regression coefficients in α_{t}.

Now, model (4.9) can be used to implement the Kalman filter equations as discussed in section 4.3. These

provide an estimate ˆα_{t }for α_{t}, which also contains the estimated coefficient for the intervention variables

β_{k}, denoted by ^{ˆ}

β_{k}. Then, the time series k after the moment of survey redesign can be adjusted for those

24

estimated discontinuities as proposed by van den Brakel et al. (2008) with

˜y_{t,k }= ˆy_{t,k }− ^{ˆ}

β_{k}. (4.10)

Here ˜y_{t,k }is the series where the quantified discontinuities are eliminated.

4.2.4 Extensions

As aforementioned, model (4.9) can be extended for several reasons. One could add explanatory

variables to better explain the trend or level of the series, or add data to investigate subpopulations.

Including an explanatory variable in the intervention variable

As discussed in section 3.1, with the introduction of the ISM in 2008, the SM_{IV }was conducted parallel

to it in 2008-2010. The difference ˆy_{ISM,t }− ˆy_{SM}_{IV }_{,t}, which is the yearly difference between the ISM and

the SM_{IV }surveys, can be seen as the discontinuity between the SM_{IV }and the new ISM. It is possible

to use this difference as explanatory information in the form of a time varying intervention variable δ^{i}

t^{.}

More specific, in (4.6) δ^{i}_{t }is initially a constant after a survey is introduced, equal to 1. When including

the explanatory variable ˆy_{dif,t }= ˆy_{ISM,t }− ˆy_{SM}_{IV }_{,t }in the intervention variable, (4.6) should be adjusted

to

δ^{i}

t ^{=}

⎧

⎪_{⎨}

⎪_{⎩}

ˆy_{dif,t }τ_{i }≤ t < τ_{i}_{+1}

0 otherwise

. (4.11)

Now (4.11) can be used as the intervention variable after the introduction of the ISM survey in 2008.

Including an explanatory variable as a new variable

Another way of implementing an explanatory variable is to model it simultaneously with the target

variables and let the errors correlate with the corresponding parameter of interest. In contrast with the

the previous approach, the explanatory series could also contain a discontinuity, which in turn also can

be quantified with the approach discussed in this section. It is not necessary that the series is available

at the point a survey is redesigned, such that any correlated series can be used.

The model as described in (4.9) is applied to the original series and the explanatory series simultaneously,

such that the number of variables in (4.9) becomes k^{expl }= k + 1. The explanatory variable used in this

paper is the police series as described in section 3.2.2. An additional constraint is that the irregular

terms of the stochastic trend are correlated, meaning that in (4.9e) Q_{t }is not a diagonal matrix anymore

but contains off-diagonal values for the correlations between the explanatory variable and its correlated

target variable. The covariance matrix of the transition equation is implemented as Q_{t }= C_{t}D_{t}C^{′}

t^{, where}

D_{t }is diagonal and C_{t }is the lower triangular with ones on the diagonal and the correlations on the

other entries. This decomposition is known as the Cholesky decomposition and ensures that Q_{t }is always

positive semidefinite. Implementing the police series as an explanatory variable results in the following

25

state space model :

α_{t }= ^{(}µ_{t,}_{1}, r_{t,}_{1}, . . . , µ_{t,}_{k}_{expl}, r_{t,}_{k}_{expl}, β^{D}^{1}_{1 }, . . . , β^{D}^{1}

k

Z_{t }=

, β^{D}^{2}_{1 }, . . . , β^{D}^{2}, β

k

D3_{1 }, . . . , β^{D}^{3},

k

β_{expl}

(

I_{k}_{expl }⊗ (1, 0)|Blockdiag^{(}(1, 1, 1) ⊗ δ^{i}

t^{I}^{k}^{, δ}^{i}

t

T = Blockdiag(T_{tr}, I_{3}_{k}_{+1}), T_{tr }= I

_{k}_{expl }⊗ (

1 1

0 1

)_{′}

, (4.12a)^{))}, (4.12b)

) , (4.12c)

ϕ_{t }= (ϕ_{t,}_{1}, . . . , ϕ_{t,}_{k}, ɛ_{t,expl})^{′}, η_{t }= (0, η_{t,r,}_{1}, . . . , 0, η_{t,r,}_{k}_{expl}, 0^{′}

3k+1^{)}^{′}^{,}

(4.12d)

E(ϕ_{t}) = 0_{k}_{expl}, E(η_{t}) = 0_{2}_{k}_{expl}_{+(3}_{k}_{+1)}, (4.12e)

(

1

H_{t }= Cov(ϕ_{t}) = Diag (σ^{2}_{ϕ,}_{1}, . . . , σ^{2}

ϕ,k^{)}^{, σ}^{2 }^{)}_{ɛ,expl }, (4.12f)

n_{t}

D_{t }= Cov(η_{t}) = Diag(0, d^{2}_{r,}_{1}, . . . , 0, d^{2}_{r,}_{k}_{expl}, 0_{3}_{k}_{+1}). (4.12g)

Here, δ^{i}_{t }is defined in (4.6), such that using (4.11) also the difference ˆy_{ISM,t}−ˆy_{SM}_{IV }_{,t }can be implemented.

Recall from section 3.2.2 that the police series does not contain any sampling error, such that its irregular

term of the measurement equation is equal to ɛ_{t,expl}. Similarly, the variance of the measurement equation

of the police series is σ^{2}_{ɛ,expl}. β_{expl }refers to the coefficient of the intervention variable for the police series.

Note that in case the series would not be correlated, C_{t }would be the identity matrix and Q_{t }= D_{t}. In case

of the police series, C_{t }is an identity matrix with a coefficient c on the off-diagonal entry (total, police) of

C_{t}, where total and police refer to the series which are correlated. The correlation between the irregular

terms of the trends of the police series and the total victimization series is then computed as

Corr(η_{total}, η_{police}) =

cd_{r,total}

√

c^{2}d^{2}

,

r,total

+ d_{r,total}d_{r,police}

where η_{total }and η_{police }refer to the irregular terms in the transition equation of both series. d^{2}

r,total ^{and}

d^{2}_{r,police }are the elements in D_{t }for the total victimization and police series, respectively. Note that if

the trends of total victimization and the police series are cointegrated, then d_{r,police }→ 0, such that

Corr(η_{police}, η_{total}) → 1.

Benchmarking with series

If there is data available for separate subpopulations as well as for the national level, it might be interesting

to investigate effects on the different subpopulations. Suppose there are H subpopulations. The

victimization series considered in this paper are reported as a percentage. Then, the following relationship

between the series ˆy_{t }at the national level and the parameters estimates ˆy_{t,}_{h }at subpopulation level

must hold :

H_{∑ }_{N}_{t,}_{h}

ˆy_{t }= ˆy_{t,}_{h}, (4.13)

h=1

N_{t}

where h refers to a subpopulation in H and N_{t,}_{h }and N_{t }= ^{∑}^{H}

_{h}_{=1 }N_{t,}_{h }is the population size at time t at

26

subpopulation and national level, respectively.

Applying the models as described in section 4.2.3 separately to the H subpopulations and the series

at national level might result in inconsistent series after adjustment for discontinuities as described in

(4.10). This is because the right proportion of each subpopulation at national level is not incorporated

in the regression coefficient for the intervention variables β_{k}.

As discussed in van den Brakel and Roels (2010), there exist several methods to implement constraint

(4.13). One method is to consider the series at national and subpopulation level simultaneously and

implement (4.13) in the regression coefficients of the intervention variables. More specific, let β = Gβ

denote the transition equation for the regression coefficients of the total population and the H subpopulations,

with β = (β^{′}

National,k^{, β}^{′}_{1}_{,k}, . . . , β^{′}

_{H }_{,k}) for every intervention. Here, β_{National,k }is the regression

coefficient for breakdown k of the intervention variable at the national level and β_{1}_{,k}, . . . , β_{H }_{,k }are the

regression coefficients for breakdown k of the intervention variable at the subpopulation level. The matrix

G is defined as follows : _{⎛}

⎜

G = _{⎝ }^{O}^{k}^{×}^{k }^{f}^{′}_{t,}_{H }⊗ I

⎞

k_{⎟}_{⎠ }, (4.14)

1_{H }⊗ O_{k}_{×}_{k }I_{H }⊗ I_{k}

where O_{k}_{×}_{k }is a k × k zero matrix, f_{t,}_{H }=

(

N_{t,}_{1}

N_{t}

, . . . , ^{N}^{t,}^{H}

N_{t}

)_{′}

is a H × 1 vector with the subpopulation

proportions, 1_{H }is a column vector with ones of order H and I_{H }an identity matrix of order H .

Since the series at national level and subpopulation level are considered simultaneously in this approach,

for the gender subpopulations of the victimization series the number of variables k per time point t

becomes k^{sub }= 3 × k, since H = 2. Here, first there are k variables on the national level, and then 2k

variables for the male and female subpopulations, respectively. Then, the state space model (4.9) with

the implemented constraint (4.13) for the gender subpopulations is represented below.

α_{t }= ^{(}µ_{t,}_{1}, r_{t,}_{1}, . . . , µ_{t,}_{k}_{sub}, r_{t,}_{k}_{sub}, β^{D}^{1}_{1 }, . . . , β^{D}^{1}

k

sub^{, β}^{D}^{2}_{1 }, . . . , β^{D}^{2}

k^{sub}^{, β}^{D}^{3}_{1 }, . . . , β^{D}^{3}

k^{sub}

Z_{t }= (I_{k}_{sub }⊗ (1, 0)|(1, 1, 1) ⊗ δ^{i}_{t}I_{k}_{sub}), T = Blockdiag(T_{tr}, I_{3 }⊗ G), T_{tr }= I_{k}_{sub }⊗ ( ^{1 1}

0 1

)_{′}

, (4.15a)

) , (4.15b)

ϕ_{t }= (ϕ_{t,}_{1}, . . . , ϕ_{t,}_{k}_{sub})^{′}, η_{t }= (0, η_{t,r,}_{1}, . . . , 0, η_{t,r,}_{k}_{sub}, 0^{′}

3k^{sub}^{)}^{′}^{,}

(4.15c)

E(ϕ_{t}) = 0

_{k}_{sub}, E(η_{t}) = 0_{5}_{k}_{sub},

(4.15d)

(

1

H_{t }= Cov(ϕ_{t}) = Diag (σ

n_{t}

2_{ϕ,}_{1}, . . . , σ^{2}

ϕ,k^{)}^{, }^{1 (}^{σ}

n_{t,}_{h}

2_{ϕ,}_{k}_{+1}, . . . , σ^{2}

ϕ,k^{sub}^{)}

)

, (4.15e)

Q_{t }= Cov(η_{t}) = Diag(0, σ^{2}_{r,}_{1}, . . . , 0, σ^{2}

r,k^{sub}^{, }^{0}^{′}

3k^{sub}^{)}^{.}

(4.15f)

Here, G is defined in (4.14). Note the changed structure of H_{t}, which is due to the assumption that the

irregular terms of the measurement equation are inversely proportional to the sample size. To meet this

assumption for the series at subpopulation level, the irregular terms for the subpopulations are divided

by the sample size of the respective subpopulations, denoted by n_{t,}_{h}. In this paper, n_{t,}_{h }is approximated

27

by ^{N}^{t,}^{h}_{N}_{t }× n_{t}. It follows that once matrix (4.14) is implemented in (4.15b) the consistency requirement is

met (van den Brakel and Roels, 2010).

4.2.5 Likelihood Ratio Tests

In subsections 4.2.1 - 4.2.4 it was assumed that the irregular terms of both the measurement equation

and the transition equation are distributed independently. This means that in H_{t }and Q_{t }the variances

are independent among the breakdowns. To obtain a more parsimonious model, one could assume that

the variances of the irregular terms of either the measurement equation or the transition equation are

equal among the breakdowns. Hence,

σ^{2}_{ϕ,k }= σ^{2}_{ϕ,k}_{′ }or σ^{2}_{r,k }= σ^{2}_{r,k}_{′ }∀ k and k’ ∈ k, (4.16)

where σ^{2}_{ϕ,k }and σ^{2}_{r,k }are defined in the model descriptions in sections 4.2.1 - 4.2.4. It is possible to test

for this assumption using the classical Likelihood Ratio Test (LRT).

4.3 Kalman filter and parameter estimation

The linear Gaussian state space model (4.7) as implied by (4.9) is restated here with the necessary

assumptions for the irregular terms :

ˆy_{t }= Z_{t}α_{t }+ ϕ_{t}, ϕ_{t }^{∼}_{= }N (0, H_{t}) ,

α_{t}_{+1 }= Tα_{t }+ η_{t}, η_{t }^{∼}_{= }N (0, Q_{t}) , (4.17)

where t = 1, . . . , t.

Note that both the covariance matrices are Normally i.i.d. distributed. In this section it is initially

assumed that the initial conditions for α_{t }are α_{1 }∼ N(a_{1}, P_{1}), with a_{1 }and P_{1 }known.

4.3.1 Filtering

Using the state space model (4.17), it is possible to apply the Kalman filter to the victimization series

as invented by the Hungarian-American electrical engineer Rudolf Emil Kálmán, known for its use in

GPS and flight control systems. ”Filtering is aimed at updating our knowledge of the system as each

observation ˆy_{t }comes in”(Durbin and Koopman, 2001).

Let the information set ^{ˆ}Y_{t }denote the set of observations ˆy_{1}, . . . , ˆy_{t}. Then, the objective of the Kalman

filter is to determine the conditional distribution of α_{t}_{+1 }given ^{ˆ}Y_{t }for t = 1, . . . , t. Because all distributions

are normal, conditional distributions of subsets of variables given other subsets are also normal and the

required distribution is therefore determined by a_{t}_{+1 }= E(α_{t}_{+1}| ^{ˆ}Y_{t}) and P_{t}_{+1 }= V ar(α_{t}_{+1}| ^{ˆ}Y_{t}). Note that

28

from (4.17) it follows that α_{t }∼ N(a_{t}, P_{t}), given information set ^{ˆ}Y_{t}. Then, box (4.18) shows the filtering

equations for the Kalman filter, which are the optimal estimators of a_{t}_{+1 }and P_{t}_{+1 }given the information

set ^{ˆ}Y_{t}. The state vector estimator E(α_{t}| ^{ˆ}Y_{t}) and its error variance matrix are denoted by a_{t}_{|}_{t }and P_{t}_{|}_{t},

respectively. For details of the derivations the reader is referred to chapter 4.2 of Durbin and Koopman

(2001).

F_{t }= Z_{t}P_{t}Z^{′}_{t }+ H_{t},

a_{t}_{|}_{t }= a_{t }+ P_{t}Z^{′}

t^{F}^{−}^{1}_{t }v_{t }P_{t}_{|}_{t }= P_{t }− P_{t}Z^{′}_{t}_{F}_{−}_{1}[ ]_{′}

t ^{P}^{t}^{Z}^{t}

v_{t }= ˆy_{t }− Z_{t}a_{t},

a_{t}_{+1 }= Ta_{t}_{|}_{t}, P_{t}_{+1 }= TP_{t}_{|}_{t}T^{′ }+ Q_{t},

(4.18)

where a_{1 }and P_{1 }are known.

The recursions in box (4.18) provide optimal estimates for the conditional distribution of α_{t}, starting from

a_{1 }and P_{1 }and then recursively updating the knowledge of the system every time a new observation ˆy_{t}

comes in. v_{t }are the so-called one-step forecast errors of ˆy_{t }given the information set ^{ˆ}Y_{t}, and F_{t }= V ar(v_{t})

is the variance matrix of v_{t}. Also, v_{t}’s are sometimes called innovations since they represent the new

part of ˆy_{t }which is not known up to point t.

4.3.2 State smoothing

While filtering focuses on updating the system every time a new observation comes in and in such

only considers the information set ^{ˆ}Y_{t}, the smoothing process aims at estimating the unobserved state

variables α_{1}, . . . , α_{n }given the entire information set ^{ˆ}Y_{t}. Hence, the smoothing process also takes into

account information that becomes available after time point t. The state vector α_{t }is estimated by its

conditional mean ˆα_{t }and its error variance matrix V_{t }= V ar(α_{t }− ˆα_{t}) is also provided. Using the Kalman

recursion in box (4.18), the recursive formulas for the fixed interval smoother are given in box (4.19) :

L_{t }= T − TP_{t}Z^{′}

t^{F}^{−}^{1}_{t }Z_{t}, w_{t}_{−}_{1 }= Z^{′}

t^{F}^{−}^{1}_{t }v_{t }+ L^{′}

t^{w}^{t}^{,}

N_{t}_{−}_{1 }= Z^{′}

t^{F}^{−}^{1}_{t }Z_{t }+ L^{′}_{t}N_{t}L_{t}, such that

ˆα_{t }= a_{t }+ P_{t}w_{t}_{−}_{1},

V_{t }= P_{t }− P_{t}N_{t}_{−}_{1}P_{t},

(4.19)

where t = t, . . . , 1, w_{t }= 0 and N_{t }= 0.

Note that in (4.19) we proceed backwards from t to 1 to obtain ˆα_{t}, whereas in box (4.18) we proceeded

forwards updating the knowledge of the system every time a new observation comes in. For the details

of the derivation of (4.19) and alternative approaches to smoothing, the reader is referred to chapter 4.3

of Durbin and Koopman (2001).

29