Nonlinear Time Series Theory Methods And Applications With R Examples 1st Edition Randal Douc

Nonlinear Time Series Theory Methods And Applications With R Examples 1st Edition Randal Douc
Nonlinear Time Series Theory Methods And Applications With R Examples 1st Edition Randal Douc
Nonlinear Time Series Theory Methods And Applications With R Examples 1st Edition Randal Douc

1. Nonlinear Time Series Theory Methods And
Applications With R Examples 1st Edition Randal
Douc download
https://ebookbell.com/product/nonlinear-time-series-theory-
methods-and-applications-with-r-examples-1st-edition-randal-
douc-4630588
Explore and download more ebooks at ebookbell.com

2. Here are some recommended products that we believe you will be
interested in. You can click the link to download.
Nonlinear Time Series Analysis In The Geosciences Applications In
Climatology Geodynamics And Solarterrestrial Physics 1st Edition
Alexander Gluhovsky Auth
https://ebookbell.com/product/nonlinear-time-series-analysis-in-the-
geosciences-applications-in-climatology-geodynamics-and-
solarterrestrial-physics-1st-edition-alexander-gluhovsky-auth-4521156
Nonlinear Time Series Analysis Of Business Cycles 1st Costas Milas
https://ebookbell.com/product/nonlinear-time-series-analysis-of-
business-cycles-1st-costas-milas-4627866
Nonlinear Time Series Analysis 2nd Edition Holger Kantz Thomas
Schreiber
https://ebookbell.com/product/nonlinear-time-series-analysis-2nd-
edition-holger-kantz-thomas-schreiber-4769506
Nonlinear Time Series Analysis With R 1st Edition Ray Huffaker
https://ebookbell.com/product/nonlinear-time-series-analysis-
with-r-1st-edition-ray-huffaker-6855674

3. Nonlinear Time Series Analysis Ruey S Tsay Rong Chen
https://ebookbell.com/product/nonlinear-time-series-analysis-ruey-s-
tsay-rong-chen-7202106
Nonlinear Time Series Semiparametric And Nonparametric Methods 1st
Edition Jiti Gao
https://ebookbell.com/product/nonlinear-time-series-semiparametric-
and-nonparametric-methods-1st-edition-jiti-gao-896860
Nonlinear Time Series Nonparametric And Parametric Methods 1st Edition
Jianqing Fan
https://ebookbell.com/product/nonlinear-time-series-nonparametric-and-
parametric-methods-1st-edition-jianqing-fan-1084970
Nonlinear Time Series Analysis In The Geosciences Applications In
Climatology Geodynamics And Solarterrestrial Physics Proof Reik Donner
https://ebookbell.com/product/nonlinear-time-series-analysis-in-the-
geosciences-applications-in-climatology-geodynamics-and-
solarterrestrial-physics-proof-reik-donner-927898
Applied Nonlinear Time Series Analysis Applications In Physics
Physiology And Finance 1st Edition Michael Small
https://ebookbell.com/product/applied-nonlinear-time-series-analysis-
applications-in-physics-physiology-and-finance-1st-edition-michael-
small-1103188

6. Randal Douc
Eric Moulines
David Stoffer
Nonlinear
Time Series
Theory, Methods, and
Applications with R Examples

8. CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Series Editors
Francesca Dominici, Harvard School of Public Health, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Statistical Theory: A Concise Introduction
F. Abramovich and Y. Ritov
Practical Multivariate Analysis, Fifth Edition
A. Afifi, S. May, and V.A. Clark
Practical Statistics for Medical Research
D.G. Altman
Interpreting Data: A First Course
in Statistics
A.J.B. Anderson
Introduction to Probability with R
K. Baclawski
Linear Algebra and Matrix Analysis for
Statistics
S. Banerjee and A. Roy
Statistical Methods for SPC and TQM
D. Bissell
Bayesian Methods for Data Analysis,
Third Edition
B.P. Carlin and T.A. Louis
Second Edition
R. Caulcutt
The Analysis of Time Series: An Introduction,
Sixth Edition
C. Chatfield
Introduction to Multivariate Analysis
C. Chatfield and A.J. Collins
Problem Solving: A Statistician’s Guide,
Second Edition
C. Chatfield
Statistics for Technology: A Course in Applied
Statistics, Third Edition
C. Chatfield
Bayesian Ideas and Data Analysis: An
Introduction for Scientists and Statisticians
R. Christensen, W. Johnson, A. Branscum,
and T.E. Hanson
Modelling Binary Data, Second Edition
D. Collett
Modelling Survival Data in Medical Research,
Second Edition
D. Collett
Introduction to Statistical Methods for
Clinical Trials
T.D. Cook and D.L. DeMets
Applied Statistics: Principles and Examples
D.R. Cox and E.J. Snell
Multivariate Survival Analysis and Competing
Risks
M. Crowder
Statistical Analysis of Reliability Data
M.J. Crowder, A.C. Kimber,
T.J. Sweeting, and R.L. Smith
An Introduction to Generalized
Linear Models, Third Edition
A.J. Dobson and A.G. Barnett
Nonlinear Time Series: Theory, Methods, and
Applications with R Examples
R. Douc, E. Moulines, and D. Stoffer
Introduction to Optimization Methods and
Their Applications in Statistics
B.S. Everitt
Extending the Linear Model with R:
Generalized Linear, Mixed Effects and
Nonparametric Regression Models
J.J. Faraway
A Course in Large Sample Theory
T.S. Ferguson
Multivariate Statistics: A Practical Approach
B. Flury and H. Riedwyl
Readings in Decision Analysis
S. French
Markov Chain Monte Carlo:
Stochastic Simulation for Bayesian Inference,
Second Edition
D. Gamerman and H.F. Lopes
Bayesian Data Analysis, Third Edition
A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson,
A. Vehtari, and D.B. Rubin
Multivariate Analysis of Variance and
Repeated Measures: A Practical Approach for
Behavioural Scientists
D.J. Hand and C.C. Taylor

9. Practical Data Analysis for Designed Practical
Longitudinal Data Analysis
D.J. Hand and M. Crowder
Logistic Regression Models
J.M. Hilbe
Richly Parameterized Linear Models:
Additive, Time Series, and Spatial Models
Using Random Effects
J.S. Hodges
Statistics for Epidemiology
N.P. Jewell
Stochastic Processes: An Introduction,
Second Edition
P.W. Jones and P. Smith
The Theory of Linear Models
B. Jørgensen
Principles of Uncertainty
J.B. Kadane
Graphics for Statistics and Data Analysis with R
K.J. Keen
Mathematical Statistics
K. Knight
Nonparametric Methods in Statistics with SAS
Applications
O. Korosteleva
Modeling and Analysis of Stochastic Systems,
Second Edition
V.G. Kulkarni
Exercises and Solutions in Biostatistical Theory
L.L. Kupper, B.H. Neelon, and S.M. O’Brien
Exercises and Solutions in Statistical Theory
L.L. Kupper, B.H. Neelon, and S.M. O’Brien
Design and Analysis of Experiments with SAS
J. Lawson
A Course in Categorical Data Analysis
T. Leonard
Statistics for Accountants
S. Letchford
Introduction to the Theory of Statistical
Inference
H. Liero and S. Zwanzig
Statistical Theory, Fourth Edition
B.W. Lindgren
Stationary Stochastic Processes: Theory and
Applications
G. Lindgren
The BUGS Book: A Practical Introduction to
Bayesian Analysis
D. Lunn, C. Jackson, N. Best, A. Thomas, and
D. Spiegelhalter
Introduction to General and Generalized
Linear Models
H. Madsen and P. Thyregod
Time Series Analysis
H. Madsen
Pólya Urn Models
H. Mahmoud
Randomization, Bootstrap and Monte Carlo
Methods in Biology, Third Edition
B.F.J. Manly
Introduction to Randomized Controlled
Clinical Trials, Second Edition
J.N.S. Matthews
Statistical Methods in Agriculture and
Experimental Biology, Second Edition
R. Mead, R.N. Curnow, and A.M. Hasted
Statistics in Engineering: A Practical Approach
A.V. Metcalfe
Beyond ANOVA: Basics of Applied Statistics
R.G. Miller, Jr.
A Primer on Linear Models
J.F. Monahan
Applied Stochastic Modelling, Second Edition
B.J.T. Morgan
Elements of Simulation
B.J.T. Morgan
Probability: Methods and Measurement
A. O’Hagan
Introduction to Statistical Limit Theory
A.M. Polansky
Applied Bayesian Forecasting and Time Series
Analysis
A. Pole, M. West, and J. Harrison
Statistics in Research and Development,
Time Series: Modeling, Computation, and
Inference
R. Prado and M. West
Introduction to Statistical Process Control
P. Qiu
Sampling Methodologies with Applications
P.S.R.S. Rao

10. A First Course in Linear Model Theory
N. Ravishanker and D.K. Dey
Essential Statistics, Fourth Edition
D.A.G. Rees
Stochastic Modeling and Mathematical
Statistics: A Text for Statisticians and
Quantitative
F.J. Samaniego
Statistical Methods for Spatial Data Analysis
O. Schabenberger and C.A. Gotway
Large Sample Methods in Statistics
P.K. Sen and J. da Motta Singer
Decision Analysis: A Bayesian Approach
J.Q. Smith
Analysis of Failure and Survival Data
P. J. Smith
Applied Statistics: Handbook of GENSTAT
Analyses
E.J. Snell and H. Simpson
Applied Nonparametric Statistical Methods,
Fourth Edition
P. Sprent and N.C. Smeeton
Data Driven Statistical Methods
P. Sprent
Generalized Linear Mixed Models:
Modern Concepts, Methods and Applications
W. W. Stroup
Survival Analysis Using S: Analysis of
Time-to-Event Data
M. Tableman and J.S. Kim
Applied Categorical and Count Data Analysis
W. Tang, H. He, and X.M. Tu
Elementary Applications of Probability Theory,
Second Edition
H.C. Tuckwell
Introduction to Statistical Inference and Its
Applications with R
M.W. Trosset
Understanding Advanced Statistical Methods
P.H. Westfall and K.S.S. Henning
Statistical Process Control: Theory and
Practice, Third Edition
G.B. Wetherill and D.W. Brown
Generalized Additive Models:
An Introduction with R
S. Wood
Epidemiology: Study Design and
Data Analysis, Third Edition
M. Woodward
Experiments
B.S. Yandell

11. Texts in Statistical Science
Randal Douc
Telecom SudParis
Evry, France
Eric Moulines
Telecom ParisTech
Paris, France
David Stoffer
University of Pittsburgh
Pennsylvania, USA
Randal Douc
Eric Moulines
David Stoffer
Nonlinear
Time Series
Theory, Methods, and
Applications with R Examples

12. CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2014 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
Version Date: 20131111
International Standard Book Number-13: 978-1-4665-0225-3 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information stor-
age or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copy-
right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222
Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro-
vides licenses and registration for a variety of users. For organizations that have been granted a pho-
tocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com

13. Contents
Preface
Frequently Used Notation
I Foundations
1 Linear Models
1.1 Stochastic processes
1.2 The covariance world
1.2.1 Second-order stationary processes
1.2.2 Spectral representation
1.2.3 Wold decomposition
1.3 Linear processes
1.3.1 What are linear Gaussian processes?
1.3.2 ARMA models
1.3.3 Prediction
1.3.4 Estimation
1.4 The multivariate cases
1.4.1 Time domain
1.4.2 Frequency domain
1.5 Numerical examples
Exercises
2 Linear Gaussian State Space Models
2.1 Model basics
2.2 Filtering, smoothing, and forecasting
2.3 Maximum likelihood estimation
2.3.1 Newton–Raphson
2.3.2 EM algorithm
2.4 Smoothing splines and the Kalman smoother
2.5 Asymptotic distribution of the MLE
2.6 Missing data modications
2.7 Structural component models
2.8 State-space models with correlated errors
2.8.1 ARMAX models
vii

14. viii CONTENTS
2.8.2 Regression with autocorrelated errors
Exercises
3 Beyond Linear Models
3.1 Nonlinear non-Gaussian data
3.2 Volterra series expansion
3.3 Cumulants and higher-order spectra
3.4 Bilinear models
3.5 Conditionally heteroscedastic models
3.6 Threshold ARMA models
3.7 Functional autoregressive models
3.8 Linear processes with innite variance
3.9 Models for counts
3.9.1 Integer valued models
3.9.2 Generalized linear models
3.10 Numerical examples
Exercises
4 Stochastic Recurrence Equations
4.1 The Scalar Case
4.1.1 Strict stationarity
4.1.2 Weak stationarity
4.1.3 GARCH(1, 1)
4.2 The Vector Case
4.2.1 Strict stationarity
4.2.2 Weak stationarity
4.2.3 GARCH(p, q)
4.3 Iterated random function
4.3.1 Strict stationarity
4.3.2 Weak stationarity
Exercises
II Markovian Models
5 Markov Models: Construction and Denitions
5.1 Markov chains: Past, future, and forgetfulness
5.2 Kernels
5.3 Homogeneous Markov chain
5.4 Canonical representation
5.5 Invariant measures
5.6 Observation-driven models
5.7 Iterated random functions
5.8 MCMC methods
5.8.1 Metropolis-Hastings algorithm
5.8.2 Gibbs sampling

15. CONTENTS ix
Exercises
6 Stability and Convergence
6.1 Uniform ergodicity
6.1.1 Total variation distance
6.1.2 Dobrushin coefcient
6.1.3 The Doeblin condition
6.1.4 Examples
6.2 V-geometric ergodicity
6.2.1 V-total variation distance
6.2.2 V-Dobrushin coefcient
6.2.3 Drift and minorization conditions
6.2.4 Examples
6.3 Some proofs
6.4 Endnotes
Exercises
7 Sample Paths and Limit Theorems
7.1 Law of large numbers
7.1.1 Dynamical system and ergodicity
7.1.2 Markov chain ergodicity
7.2 Central limit theorem
7.3 Deviation inequalities for additive functionals
7.3.1 Rosenthal type inequality
7.3.2 Concentration inequality
7.4 Some proofs
Exercises
8 Inference for Markovian Models
8.1 Likelihood inference
8.2 Consistency and asymptotic normality of the MLE
8.2.1 Consistency
8.2.2 Asymptotic normality
8.3 Observation-driven models
8.4 Bayesian inference
8.5 Some proofs
8.6 Endnotes
Exercises
III State Space and Hidden Markov Models
9 Non-Gaussian and Nonlinear State Space Models
9.1 Denitions and basic properties
9.1.1 Discrete-valued state space HMM
9.1.2 Continuous-valued state-space models

16. x CONTENTS
9.1.3 Conditionally Gaussian linear state-space models
9.1.4 Switching processes with Markov regimes
9.2 Filtering and smoothing
9.2.1 Discrete-valued state-space HMM
9.2.2 Continuous-valued state-space HMM
9.3 Endnotes
Exercises
10 Particle Filtering
10.1 Importance sampling
10.2 Sequential importance sampling
10.3 Sampling importance resampling
10.3.1 Algorithm description
10.3.2 Resampling techniques
10.4 Particle lter
10.4.1 Sequential importance sampling
10.4.2 Auxiliary sampling
10.5 Convergence of the particle lter
10.5.1 Exponential deviation inequalities
10.5.2 Time-uniform bounds
10.6 Endnotes
Exercises
11 Particle Smoothing
11.1 Poor man's smoother algorithm
11.2 FFBSm algorithm
11.3 FFBSi algorithm
11.4 Smoothing functionals
11.5 Particle independent Metropolis-Hastings
11.6 Particle Gibbs
11.7 Convergence of the FFBSm and FFBSi algorithms
11.7.1 Exponential deviation inequality
11.7.2 Asymptotic normality
11.7.3 Time uniform bounds
11.8 Endnotes
Exercises
12 Inference for Nonlinear State Space Models
12.1 Monte Carlo maximum likelihood estimation
12.1.1 Particle approximation of the likelihood function
12.1.2 Particle stochastic gradient
12.1.3 Particle Monte Carlo EM algorithms
12.1.4 Particle stochastic approximation EM
12.2 Bayesian analysis
12.2.1 Gaussian linear state space models

17. CONTENTS xi
12.2.2 Gibbs sampling for NLSS model
12.2.3 Particle marginal Markov chain Monte Carlo
12.2.4 Particle Gibbs algorithm
12.3 Endnotes
Exercises
13 Asymptotics of the MLE for NLSS
13.1 Strong consistency of the MLE
13.1.1 Forgetting the initial distribution
13.1.2 Approximation by conditional likelihood
13.1.3 Strong consistency
13.1.4 Identiability of mixture densities
13.2 Asymptotic normality
13.2.1 Convergence of the observed information
13.2.2 Limit distribution of the MLE
13.3 Endnotes
Exercises
IV Appendices
Appendix A Some Mathematical Background
A.1 Some measure theory
A.2 Some probability theory
Appendix B Martingales
B.1 Denitions and elementary properties
B.2 Limits theorems
Appendix C Stochastic Approximation
C.1 Robbins–Monro algorithm: Elementary results
C.2 Stochastic gradient
C.3 Stepsize selection and averaging
C.4 The Kiefer–Wolfowitz procedure
Appendix D Data Augmentation
D.1 The EM algorithm in the incomplete data model
D.2 The Fisher and Louis identities
D.3 Monte Carlo EM algorithm
D.3.1 Stochastic approximation EM
D.4 Convergence of the EM algorithm
D.5 Convergence of the MCEM algorithm
D.5.1 Convergence of perturbed dynamical systems
D.5.2 Convergence of the MCEM algorithm

18. xii CONTENTS
References
Index

19. Preface
This book is designed for researchers and students who want to acquire advanced
skills in nonlinear time series analysis and their applications. Before reading this
text, we suggest a solid knowledge of linear Gaussian time series, for which there
are many texts. At the advanced level, texts that cover both the time and frequency
domains are1994),1991), and1996). At the
intermediate level, we mention1994),2010), and
and Stoffer2011), which cover both the time and frequency domains, and
Jenkins1970), which covers primarily the time domain.2012)
is an advanced text on the statistical theory of linear state space systems. There are a
number of texts that cover time series at a more introductory level, but the material
covered in this text requires at least an intermediate level of understanding of the
time domain.
While it is not sensible to view statistics simply as a branch of mathematics, we
believe that statistical modeling and inference need to be rmly grounded in theory.
Although we avoid delving into sophisticated mathematical derivations, most of the
statements of the book are rigorously established. The reader is therefore expected
to have some background in measure theory (covering the construction of the mea-
sure and Lebesgue integrals), and in probability theory (including conditional ex-
pectations, the construction of discrete time stochastic processes and martingales).
Examples of courses covering this material are1953),1995),
Shiryaev1996), and2010), among many others. Although we constantly
use measure-theoretic concepts and notations, nothing excessively deep is used. An
introduction to discrete state space Markov chains is clearly a plus, but is not needed.
The book represents a biased selection of topics in nonlinear time series, reect-
ing our own inclinations toward state-space representations. Our focus on principles
is intended to provide readers with a solid background to craft their own stochas-
tic models, numerical methods, and software, and to be able to assess the advantages
and disadvantages of different approaches. We do not believe in pulling mathematical
formulas out of thin air, or establishing a catalog of models and methods. Of course,
this attitude reects our mathematical orientation and our willingness to postpone the
statistical discussion to pay attention to rigorous theoretical foundations.
There are a number of texts that cover nonlinear and non-Gaussian models from
a variety of points-of-view. Because nancial series tend to be nonlinear, there are
many texts that focus primarily on nance such as2002),
(2000), and2005). The text by ¨asvirta et al.2011), while focusing primar-
ily on nance, is a rather comprehensive and approachable text on the subject. Other
xiii

20. xiv PREFACE
texts that present general statistical approaches to nonlinear time series models are
Fan and Yao2003) and2007), which take a nonparametric or semiparametric
smoothing approach,1988), which focuses on nonlinear models and spec-
tral analysis for nonstationary processes, and1983), which introduces thresh-
old models.1990) and2004) take a dynamical systems
approach and present a wide array of nonlinear time series models. Two other texts
that focus primarily on a state-space approach to nonlinear and non-Gaussian time
series are1996) and the second part of
(2012).2009),2008) and2010) present
a state-space approach to modeling linear and nonlinear time series at an introduc-
tory level.2010) could serve as a supplement for readers seeking a more
gentle initial approach to the subject.
We are agnostic about the nature of statistical inference. The reader must def-
initely look elsewhere for a philosophical discussion of the relative merits of fre-
quentist versus Bayesian inference. Our belief is that nonlinear time series generally
benet from analysis using a variety of frequentist and Bayesian methods. These
different perspectives strengthen the conclusions rather than contradict one another.
Our hope is to acquaint readers with the main principles behind nonlinear time se-
ries models without overwhelming them with difcult mathematical developments.
To keep the book length within acceptable limits, we have avoided the use of so-
phisticated probabilistic arguments, which underlie most of the recent developments
of continuous state space Markov chains and sequential Monte Carlo methods. For
the statistical part, we cover mostly the basics; other important concepts like the lo-
cal asymptotic normality (e.g.,,), empirical process
techniques (e.g.,,), cointegration (e.g.,,), multivariate
time series (e.g.,,, ¨utkepohl,), model selection, semiparametric
and nonparametric inference (e.g.,,), and so on, may be found in
other texts. We are, of course, responsible for any and all mistakes, misconceptions
and omissions.
Although there is a logical progression through the chapters, the three parts can
be studied independently. Some chapters within each part may also be read as in-
dependent surveys. Several chapters highlight recent developments such as explicit
rate of convergence of Markov chains (we use the techniques outlined in
Mattingly,
tial Monte Carlo techniques (covering for example the recently introduced particle
Markov chain Monte Carlo methods found in,).
Any instructor contemplating a one-semester course based on this book will have
to decide which chapters to cover and which to omit. The rst part can be seen as a
crash course on “classical” time series, with a special emphasis on linear state space
models and a rather detailed coverage on random coefcient autoregressions, cover-
ing both ARCH and GARCH models. The second part is a self-contained introduc-
tion to Markov chain, discussing stability, the existence of a stationary distribution,
ergodicity, limit theorems and statistical inference. Many examples are provided with
the objective to develop empirical skills. We have already covered parts I and II in
a fast-paced one semester advanced master level course. Part III is a self-contained

21. PREFACE xv
account of nonlinear state space and sequential Monte Carlo methods. It is an ele-
mentary introduction to nonlinear state space modeling and sequential Monte Carlo,
but it touches on many current topics in this eld, from the theory of statistical in-
ference to advanced computational methods. This has been used as a support to an
advanced course on these methods, and can be used by readers who want to have an
introduction to this eld before studying more specialized texts such as
(2004) or2010).
As with any textbook, the exercises are nearly as important as the main text.
Statistics is not a spectator sport, so the book contains more than 200 exercises to
challenge the readers. Most problems merely serve to strengthen intellectual mus-
cles strained by the introduction of new theory; some problems extend the theory in
signicant ways.
We acknowledge the help of Julien Cornebise and Fredrik Lindsten who partici-
pated in the writing of the text and contributed to
III. Julien also helped us considerably in the development ofRcode. We are also in-
debted to Pierre Priouret for suggesting various forms of improvement in the presen-
tation, layout, and so on, as well as helping us track typos and errors. We are grateful
to Christophe Andrieu, Pierre Del Moral, and Arnaud Doucet, who generously gave
us some of their time to help to decipher many of the intricacies of the particle lters.
This work would have not been possible without the continuous support of our col-
leagues and friends Olivier Capp´e, Gersende Fort, Jimmy Olsson, Franc¸ois Roueff,
and Philippe Soulier, who provided various helpful insights and comments. We also
acknowledge Hedibert Freitas Lopes and Fredrik Lindsten for distributing code that
became the basis of some of theRscripts used in. Finally, we thank
John Kimmel for his support and enduring patience.
R. Douc thanks Telecom SudParis for a six month sabbatical in 2012 to work on
the text. E. Moulines thanks Telecom ParisTech for giving him the opportunity to
work on this project. D.S. Stoffer thanks the U.S. National Science Foundation and
Telecom ParisTech for partial support during the preparation of this manuscript. In
addition, most of the time spent on the text was during a visiting professorship at
the Booth School of Business, University of Chicago, and their support is gratefully
acknowledged.
The webpage for the text,, contains the Rscripts
used in the examples and other useful information such as errata.
Paris Randal Douc
Paris Eric Moulines
Chicago David S. Stoffer
September, 2013

23. FrequentlyUsedNotation
Sets and Numbers
•N: the set of natural numbers including zero,N=f0;1;2;:::g.
•N

: the set of natural numbers excluding zero,N

=f1;2;:::g.
•Z: the set of relative integers,Z=f0;1;2;:::g.
•R: the set of real numbers.
•R
d
: Euclidean space consisting of all column vectorsx= (x1; :::;xd)
0
.
•¯R: the extended real line, i.e.,R[ f¥;+¥g.
•C: the set of complex numbers.
•\;
T
: intersection.
•[;
S
: union.
•A
c
: the complement ofA.
•BnA:B\A
c
, the relative complement ofBinAor set difference.
•A B: symmetric difference of sets;(BnA)[(AnB).
•zorz

: the complement ofz2C.
•dxe: the smallest integer bigger than or equal tox.
•bxc: the largest integer smaller than or equal tox.
•ab: convolution; for sequencesa=fa(n);n2Zgandb=fb(n);n2Zg,ab
denotes the convolution ofaandb, dened formally byab=åka(k)b(nk).
Metric space
•(X;d): a metric space.
•(x;r): the open ball of radiusr>0 centerd inx, B(x;r) =fy2X:d(x;y)<rg:
•U: closure of the setUX.
•U
o
: interior of the setUX.
•¶U: boundary of the setUX.
Binary relations
•a^b: the minimum ofaandb.
•a_b: the maximum ofaandb.
xvii

24. xviii FREQUENTLY USED NOTATION
•a(n)b(n): the ratio of the two sides is bounded from above and below by posi-
tive constants that do not depend onn.
•a(n)b(n): the ratio of the two sides converges to one.
Vectors, matrices
•Md(R)(resp.Md(C)): the set ofddmatrices with real (resp. complex) coef-
cients.
•kjMkj: operator norm; forM2Md(C), andk kany norm onC
d
,
kjMkj=sup

kMxk
kxk
;x2C
d
;x6=0

:
•d:ddidentity matrix.
•11= (1;:::;1)
0
:d1 vector whose entries are all equal to 1.
•AB: Kronecker product; letAandBbemnandpqmatrices, respectively.
The Kronecker product ofAwithBis thempnqmatrix whose(i;j)th block
is thepq Ai;jB, whereAi;jis the(i;j)th element ofA. Note that the Kronecker
product is associative(AB)C=A(BC)and(AB)(CB) = (ACBD)
(for matrices with compatible dimensions).
• (A): vectorization of a matrix; letAbe anmnmatrix, then Vec(A)is the
(mn1)vector obtained fromAby stacking the columns ofA(from left to right).
Note that Vec(ABC) = (C
T
A)Vec(B).
Functions
•1A: indicator function with1A(x) =1 ifx2Aand 0 otherwise.1fAgis used ifA
is a composite statement
•f
+
: the positive part of the functionf, i.e.,f
+
(x) =f(x)_0,
•f

: the negative part of the functionf, i.e.,f

(x) =(f(x)^0).
•f
1
(A): inverse image of the setAbyf.
• (f): the oscillation seminorm; forfa real valued function onX,jfj
¥=
supff(x):x2Xgit is the supremum norm dened as
osc(f) =sup
(x;y)2XX
jf(x)f(y)j=2 inf
c2R
jfcj
¥:
Measures
Let(X;X)be a measurable space.
•Xis a topological space (in particular a metric space) thenXis always taken
to be the Borel sigma-eld generated by the topology ofX. IfX=¯R
d
, its Borel
sigma-eld is denoted byFb

¯R
d

.

25. FREQUENTLY USED NOTATION xix
•dx: Dirac measure with mass concentrated onx, i.e.,dx(A) =1 ifx2Aand 0
otherwise.
• R
d
.
•M(X): the set of nite signed measures on the measurable space(X;X).
•M+(X): the set of measures on the measurable space(X;X).
•M1(X): the set of probability measures on(X;X).
•M0(X): the set of nite signed measuresxon(X;X)satisfyingx(X) =0.
•mn:mis absolutely continuous with respect ton.
Function spaces
Let(X;X)be a measurable space.
•F(X;X): the vector space of measurable functions from(X;X)to(¥;¥).
•F+(X;X): the cone of measurable functions from(X;X)to[0;¥].
•Fb(X;X): the subset ofF(X;X)of bounded functions.
•x(f): for anyx2M(X)andf2Fb(X;X),x(f) =
R
fdx.
•x2M(X): denes a linear functional on the Banach space(Fb(X;X);j j
¥). We
use the same notation for the measure and for the functional.
• b(X): the space of all bounded continuous real functions dened on a topological
spaceX.
•L
p
(m): the space of measurable functionsfsuch that
R
jfj
p
dm<¥.
•
p
(m): the space of classes ofm-equivalent functions inL
p
(m). Iff2L
p
(m),
kfkp= (
R
jfj
p
dm)
1=p
wheref2f. When no confusion is possible, we will identify
fand anyf2f.
Probability space
Let(W;A;P)be a probability space.
•E[X]: expectation of random variableXwith respect to the probability measure
P.
• [X]variance of random variableXwith respect to the probability measureP
• (X;Y): covariance of the random variablesXandY.
•P(AjF): conditional probability ofAgivenF, a sub-s-eldF, andA2 A.
•E

X

F

: conditional expectation ofXgivenFas dened above.
•LP(X): the law ofXunderP.
•Xn
P
=)XorXn)PX: the sequence of random variablesfXngconverges toXin
distribution underP.
•Xn
P
!XorXn!PX: the sequence of random variablesfXngconverges toXin
probability underP.

26. xx FREQUENTLY USED NOTATION
•Xn
P-a.s.
!XorXn!P-a.s.X: the sequence of random variablesfXngconverges toX
P-almost surely (P-a:s:).
•X
d
=YorX=dY:Xis stochastically equal toY; i.e.,LP(X) =LP(Y)
Usual distributions
•(m;s
2
): Normal distribution with meanmand variances
2
.
•g(x;m;S): The Gaussian density in variablexwith meanmand (co)varianceS;
see (1.36).
•(a;b): uniform distribution on[a;b].
•c
2
: chi-square distribution.
•c
2
n: chi-square distribution withndegrees of freedom.
White noise types
•ZtWN(0;s
2
): The sequencefZt;t2 T gis an uncorrelated sequence with mean
zero and variances
2
. The actual setTwill be apparent from the context. Called
second-order white noise,second-order noise,weak white noise,weak noise, or
simplywhite noise.
•Ztiid(0;s
2
): The sequencefZt;t2 T gis an i.i.d. sequence with mean zero and
variances
2
. The actual setTwill be apparent from the context. Calledstrong
white noise,strong noise,i.i.d. noise, orindependent white noise.
•Ztiid N(0;s
2
): The sequencefZt;t2 T gis an i.i.d. sequence of Gaussian ran-
dom variables with mean zero and variances
2
. The actual setTwill be apparent
from the context. CalledGaussian noiseornormal noise.

27. PartI
Foundations
1

29. Chapter 1
LinearModels
In this chapter, we briey review some aspects of stochastic processes and linear
time series models in both the time and frequency domains. The chapter can serve as
a review of stationarity, linearity and Gaussianity that provides a foundation on which
to build a course on nonlinear time series analysis. Our discussions are brief and are
meant only to establish a baseline of material that is necessary for comprehension of
the material presented in this text.
1.1 Stochastic processes
The primary objective of time series analysis is to develop mathematical models that
allow plausible descriptions for sample data. In order to provide a statistical setting
for describing the character of data that seemingly uctuate in a random fashion over
time, we assume a time series can be dened as a collection of random elements,
dened on some probability space(W;F;P)and taking value in some state-spaceX,
indexed according to the order they are obtained in time,
fXt;t2 T g: (1.1)
Timettakes values in setT, which can be discrete or continuous. In this book, we
will takeTto be the integersZ=f0;1;2;:::;gor some subset of the integers
such as the non-negative integersN=f0;1;2;:::g. For example, we may consider a
time series as a sequence of random variables,fXt;t2Ng=fX0;X1;X2;:::g, where
the random variableX0denotes the initial value taken by the series,X1denotes the
value at the rst time period,X2denotes the value for the second time period, and so
on.
Thestate-spaceXis the space in which the time series takes its values. Formally,
the state-space should be a measurable space,(X;X)whereXis as-eld. In many
instances,X=RorX=R
d
, if the observations are scalar or vector-valued. In this
case,X=B(R)orX=B(R
d
), the Borels-elds. In some examples, the obser-
vations can be discrete; for example, the observations are integersX=Nif we are
dealing with time-series of counts.
Denition 1.1 (Stochastic process).A collection of random elements X=fXt;t2
T gdened on a probability space(W;F;P)and taking value in a measurable space
(X;X)is referred to as a stochastic process.
3

30. 4 1. LINEAR MODELS
A stochastic process is therefore a function of two argumentsX:T W!X,
(t;w)7!Xt(w). For eacht2 T,Xt:w7!Xt(w)is a straightforwardF=X-measurable
random element fromWtoX, which induces a probability measure on(X;X)(re-
ferred to as the the law ofXt). For eachw2W,X(w):t7!Xt(w)is a function from
Tto the state-spaceX.
Denition 1.2 (Trajectory and path).To everyw2Wis associated a collection of
numbers (or vector of numbers), indexed byT, t7!Xt(w)representing arealization
of the stochastic process (sometimes referred to as apath).
Because it will be clear from the context of our discussions, we use the term time
series whether we are referring generically to the process or to a particular realization
and make no notational distinction between the two concepts.
The distribution of a stochastic process is usually described in terms of itsnite-
dimensional distributions.
Denition 1.3 (Finite-dimensional distributions).The nite-dimensional distri-
butions of the stochastic process X are the set of all the joint distributions of the
random elements Xt1
;Xt2
;:::;Xtn
for all integer n and all n-tuples(t1;t2;:::tn)2 T
n
of distinct indices (i.e., ti6=tjfor i6=j),
m
X
t1;t2;:::;tn
(H) =P[(Xt1
;Xt2
;:::;Xtn
)2H];H2 X
n
: (1.2)
Finite dimensional distributions are enough to compute the distribution of measur-
able functions involving a nite number of random elements(Xt1
;Xt2
;:::;Xtn
), such
as, e.g., the product ofXt1
Xt2
:::Xt
k
or the maximum, max(Xt1
;Xt2
;:::;Xt
k
). But in
the sequel we will have to consider quantities involving an innite number of ran-
dom variables, like the distribution of the maxt2TXtor limits such as limt!¥Xt. Of
course, a priori, computing the distribution of such quantities require us to go beyond
anynitedimensional distributions. The following theorem states that the nite-
dimensional distributions specify the innite-dimensional distribution uniquely.
Theorem 1.4 (Stochastic processes / nite-dimensional distributions).Let X and
Y be twoX-valued stochastic processes indexed byT. Then X and Y have the same
distribution if and only if all their nite-dimensional distributions agree.
The nite-dimensional distributions of a given stochastic processXare not an
arbitrary set of distributions. Consider a collection of distinct indicest1;t2;:::tn2 T,
and corresponding measurable setsB1;B2;:::;Bn2 X. Then, for any further index
tn+12 Tdistinct fromt1;:::;tn,
m
X
t1;t2;:::;tn
(B1B2 Bn) =m
X
t1;t2;:::;tn;tn+1
(B1B2 BnX):(1.3)
Similarly, ifsis a permutation of the setf1;:::;kg,
m
X
t1;t2;:::;tn
(B1B2 Bn) =m
X
t
s(1)
;t
s(2)
;:::;t
s(n)
(B
s(1)B
s(2) B
s(n)):(1.4)
Such relations are referred to asconsistencyrelations. The nite dimensional distri-
butions of a stochastic process necessarily satisfy (1.3) and (1.4). The Kolmogorov
Existence Theorembasically states the converse.

31. 1.2. THE COVARIANCE WORLD 5
Theorem 1.5 (Kolmogorov existence theorem).Let
M=

mt1;t2;:::;tn
:n2N

;(t1;t2;:::;tn)2 T
n
;with ti6=tjfor i6=j

(1.5)
be a system of nite-dimensional distributions satisfying the two consistency condi-
tions (1.3) and (1.4). Then, there exist a probability space (W;F;P)and anX-valued
stochastic process X, havingMas its nite dimensional distributions.
Proof.See for example1999, Theorem 36.2).
The proof is constructive. We can setW=X
T
=Õt2TXthe set of all possible
pathsfxt;t2 T g(for allt2 T, the coordinatext2X). Thes-algebraFcan be
identied withs-algebra generated by the cylinders, dened as
C=

fxt;t2 T g:(xt1
;xt2
;:::;xt
k
)2H

;H2 X
k
;k2N: (1.6)
In this case, the random elementsXtare the coordinate projections, forw=fxt;t2
T g,Xt(w) =xt.
Denition 1.6 (Strict stationarity).A process is said to bestrictly stationaryif
fXt1
;Xt2
;:::;Xtn
g
d
=fXt1+h;Xt2+h;:::;Xtn+hg; (1.7)
for all n2N

, all time points(t1;t2;:::;tn)2 T
n
, and all time shifts h2Z.
A trivial example of a strictly stationary process is one wherefXt;t2Zgis
i.i.d. (where i.i.d. stands for independent and identically distributed). In addition, it
is easily shown, using characteristic functions, that iffXtgis i.i.d., then a nitetime
invariant linear lterofXt, sayYt=å
k
j=k
ajXtj, where(a0;a1;:::;ak)2R
2k+1
,
is strictly stationary (see).
1.2 The covariance world
1.2.1 Second-order stationary processes
As mentioned above, a stochastic process is strictly stationary if its nite-dimensional
distributions are invariant under time-shifts. Rather than imposing conditions on all
possible distributions, a milder version imposes conditions only on the rst two mo-
ments of the series. First, we make the following denitions. Unless stated otherwise,
the state space is taken to beX=R.
Denition 1.7 (Mean function).Themean functionis dened as
mt=E[Xt] (1.8)
provided thatE[jXtj]<¥, whereEdenotes the expectation operator.
Denition 1.8 (Autocovariance function).Assume that for all t,E

X
2
t

<¥. The
autocovariance functionis dened as the second moment product
g(s;t) =Cov(Xs;Xt) =E[(Xsms)(Xtmt)]; (1.9)
for all s and t. Note thatg(s;t) =g(t;s).

32. 6 1. LINEAR MODELS
If a time series is strictly stationary, then all of the multivariate distribution func-
tions for subsets of variables must agree with their counterparts in the shifted set
for all values of the shift parameterh. For example, whenn=1, (1.7) implies that
PfXsxg=PfXtxgfor any time pointssandt. If, in addition, the mean function,
mt, exists, (1.7) implies that ms=mtfor allsandt, and hencemtmust be constant.
Whenn=2, we can write (1.7) as PfXsx1;Xtx2g=PfXs+hx1;Xt+hx2g
for any time pointssandtand shifth. Thus, if the variance function of the pro-
cess exists, this implies that the autocovariance function of the seriesXtsatises
g(s;t) =g(s+h;t+h)for allsandtandh. We may interpret this result by saying
the autocovariance function of the process depends only on the time difference be-
tweensandt, and not on the actual times. These considerations lead to the following
denition.
Denition 1.9 (Second-order stationarity).A second-order or weaklystationary
time series,fXt;t2Zg, is a nite variance process such that
(i) mt, dened in (1.8) is constant and does not depend on
time t, and
(ii) g(s;t), dened in (1.9) depends on s and t only
through their differencejstj.
For brevity, a covariance or weakly stationary series is simply called astation-
arytime series. It should be clear from the discussion following
a strictly stationary, nite variance time series is also stationary. The converse is not
true unless there are further conditions. One important case where weak stationarity
implies strict stationarity is if the time series is Gaussian (meaning all nite distri-
butions of the series are Gaussian; see). In a linear, Gaussian world,
these conditions are sufcient for inference. The idea is that one only needs to spec-
ify the mean and covariance relationships of a process to specify all nite Gaussian
distributions ala Kolmogorov's existence theorem.
Because the mean function,mt, of a stationary time series is independent of timet,
we drop the subscript and writemt=m. Also, because the autocovariance function,
g(s;t), of a stationary time series depends onsandtonly through their difference
jstj, the notation can be simplied. Lets=t+h, wherehrepresents the time shift
or lag. Then
g(t+h;t) =Cov(Xt+h;Xt) =Cov(Xh;X0) =g(h;0)
because the time difference between timest+handtis the same as the time differ-
ence between timeshand 0. Thus, the autocovariance function of a stationary time
series does not depend on the time argumentt. Hence, for convenience, the second
argument ofg(h;0)is dropped and we write
g(h):=Cov(Xt+h;Xt): (1.10)
Proposition 1.10 (Autocovariance function properties).Letg(h), as dened in
(1.10), be the autocovariance function of a stationary process. Then
(i)g(h) =g(h)for h2Z, as indicated in.

33. 1.2. THE COVARIANCE WORLD 7
(ii)jg(h)j g(0), by the Cauchy-Schwarz Inequality.
(iii)g(h)is anon-negative denite (n.n.d.) function; that is, for any set of constants
(a1;:::;an)2R
n
, time points(t1;:::;tn)2Z
n
, and any n2N

,
n
å
i=1
n
å
j=1
aig(titj)aj0: (1.11)
That(1.11) holds follows simply from the fact that for any samplefXt1
;:::;Xtn
gfrom
a stationary processfXt;t2Zg, we haveEjå
n
i=1
ai(Xti
m)j
2
0.
It follows immediately from Gn, of a
sample of sizen,fXt1
;:::;Xtn
g, from a stationary time series is symmetric and n.n.d.,
where
Gn=Cov(Xt1
;:::;Xtn
) =
2
6
6
6
4
g(0)g(t1t2)g(t1tn)
g(t2t1)g(0)g(t2tn)
.
.
.
.
.
.
.
.
.
.
.
.
g(tnt1)g(tnt2)g(0)
3
7
7
7
5
:(1.12)
As in classical statistics, it is often convenient to deal with correlation, and this
leads to the following denition.
Denition 1.11 (Autocorrelation function).Theautocorrelation function (ACF)is
dened as
r(s;t) =
g(s;t)
p
g(s;s)
p
g(t;t)
: (1.13)
The ACF measures the linear predictability of the series at timet, sayXt, using only
the valueXs. We can show easily that1r(s;t)1 using the Cauchy–Schwarz
inequality. If we can predictXtperfectly fromXsthrough a linear relationship,Xt=
a+bXs;then the correlation will be+1 whenb>0, and1 whenb<0.
If the process is stationary, the ACF is given by
r(h) =Cor(Xt+h;Xt) =
g(h)
g(0)
: (1.14)
Example 1.12 (Estimation of ACF in the stationary case).If a time series is sta-
tionary, the mean function,mt=min (1.8), is constant so that we can estimate it by
the sample mean,X=n
1
å
n
t=1
Xt. The theoretical autocovariance function, (1.9), is
estimated by the sample autocovariance function dened as follows:
bg(h) =n
1
nh
å
t=1
(Xt+hX)(XtX); (1.15)
withbg(h) =bg(h)forh=0;1;:::;n1. The sum in (1.15) runs over a restricted
range becauseXt+his not available fort+h>n. The estimator in (1.15) is preferred
to the one that would be obtained by dividing bynhbecause (1.15) is a non-
negative denite function; see (1.11).

34. 8 1. LINEAR MODELS
Thesample autocorrelation functionis dened, analogously to (1.14), as
br(h) =
bg(h)
bg(0)
: (1.16)
3
Example 1.13 (White noise).A basic building block of time series models iswhite
noise, which we denote byZt. At a basic level,fZt;t2Zgis a process of uncor-
related random variables with mean 0 and variances
2
z; in this case we will write
ZtWN(0;s
2
z). Note thatZtis stationary with ACFrz(h) =dh(0).
We will, at times, also require the noise to be i.i.d. random variables with mean
0 and variances
2
z. We shall distinguish this case by sayingindependent white noise
orstrong white noise, or by writingZtiid(0;s
2
z). A particularly useful white noise
series isGaussian white noise, wherein theZtare independent normal random vari-
ables with mean 0 and variances
2
z, or more succinctly,Ztiid N(0;s
2
z). 3
Example 1.14 (Finite time invariant linear lter).Although white noise is an un-
correlated process, correlation can be introduced by ltering the noise. For ex-
ample, supposeXtis a nite time invariant linear lter of white noise given by
Xt=å
k
j=k
ajZtj, where(a0;a1;:::;ak)2R
2k+1
, thenmt=E(Xt) =0 and
g(h) =Cov(Xt+h;Xt) =
k
åå
i;j=k
aiajCov(Zt+hi;Ztj) =s
2
zå
jjjkh
ajaj+h:
The basic idea may be extended to a lter of a general processXt, sayYt=
å
k
j=k
ajXtj; we leave it as an exercise to show that ifXtis stationary, thenYtis
stationary (see). Some important lters are the difference lterto remove
trend,Yt=XtXt1, wherek=1 anda1=0;a0=1;a1=1, and a lter used
toseasonally adjust, or remove the annual cycle in monthly data,Yt=å
6
j=6
ajXtj
whereaj=1=12 forj=0;1;:::;5 anda6=1=24. 3
The concept of a nite linear lter may be extended to an innite linear lter of
a stationary process via the L
2
completeness theorem known as theRiesz–Fischer
Theorem.
Theorem 1.15 (Riesz–Fischer).LetfUn;n2Ngbe a sequence inL
2
. Then, there
exists a U inL
2
such that Un
m.s.
!U if and only if
lim
m!¥
sup
nm
EjUnUmj
2
=0:
We now address the notion of an innite linear lter in its generality, which we
state in the following proposition.
Proposition 1.16 (Time invariant linear lter).Consider a time-invariant linear
lter dened as a convolution of the form
Yt=
¥
å
j=¥
ajXtj;
¥
å
j=¥
jajj<¥; (1.17)

35. 1.2. THE COVARIANCE WORLD 9
for each t2Z. IffXt;t2Zgis a sequence of random variables such that
sup
t2Z
E[jXtj]<¥, then the series converges absolutelyP-a:s:If, in addition,
sup
t2Z
E

jXtj
2

<¥, the series also converges in mean square to the same limit.
In particular, iffXt;t2Zgis stationary, these properties hold.
Proof.Dening the nite linear lter as
Y
n
t=
n
å
j=n
ajXtj; (1.18)
n2N, to establish mean square convergence, we need to show that, for eacht2Z,
fY
n
t;n2Nghas a mean square limit. By, it is enough to show
lim
m!¥
sup
nm
EjY
n
tY
m
t
j
2
=0: (1.19)
Forn>m>0,
EjY
n
tY
m
t
j
2
=E


å
m<jjjn
ajXtj



2
å
m<jjjn
å
mjkjn
jajjjakjjE(XtjXtk)j
sup
t2Z
E

jXtj
2


å
mjjjn
jajj

2
;
which implies (1.19), because sup
t2Z
E

jXtj
2

<¥andfajgis absolutely summable
(the second inequality follows from Cauchy–Schwarz).
Although we know that the sequencefY
n
t;n2Nggiven by (1.18) converges in
mean square, we have not established its mean square limit. If˜Ytdenotes the mean
square limit ofY
n
t, then using Fatou's Lemma,Ej˜YtYtj
2
=Eliminfn!¥j˜YtY
n
tj
2

liminfn!¥Ej˜YtY
n
tj
2
=0, which establishes thatYtis the mean square limit ofY
n
t.
It is also fairly easy to establish the fact that (1.17) exists P-a:s:We have, by the
Fubini-Tonelli theorem,
E
"
å
k2Z
jakXtkj
#
=å
k2Z
jakjE[jXtkj]sup
t2Z
E[jXtj]å
k2Z
jakj;
which is nite because sup
t2Z
E[jXtj]<¥andfajgis absolutely summable. Hence
å
¥
j=¥jajjjXtjj<¥,P-a:s:Therefore, the sequencefY
n
t;n2Ngconverges (n!¥)
absolutely toYt,P-a:s:
1.2.2 Spectral representation
It is often advantageous to analyze the repetitive or regular behavior of a linear, sta-
tionary process based on its harmonic or oscillatory nature. This idea is the founda-
tion of spectral analysis. In this case, it is easier to work with complex time series,
sayXt=Xt1+iXt2whereXt1=ReXtandXt2=ImXt; i.e., the state-space isX=C.
Stationarity follows as in the real case. Using obvious notation,Xtis sta-
tionary ifm=EXt=m1+im2is independent oftandg(h) =Cov(Xt+h;Xt) =

36. 10 1. LINEAR MODELS
E[(Xt+hm)(Xtm)

], is independent of timet, where

indicates conjugation. In
this case,g(h)may be a complex-valued function, but
with appropriate changes. That is, (i)g(h)is a Hermitian function:g(h) =g

(h),
(ii) 0 jg(h)j g(0)2R, and (iii)g(h)is n.n.d. in the sense that, for any set of
constants(a1;:::;an)2C
n
, time points(t1;:::;tn)2Z
n
, and anyn2N

,
n
å
i=1
n
å
j=1
a

ig(titj)aj0: (1.20)
The following theorem states that the autocovariance function of a weakly sta-
tionary processfXt;t2Zgis entirely determined by a nite nonnegative measure on
(p;p]. This measure is called thespectral measureoffXt;t2Zg.
Theorem 1.17 (Herglotz).A sequence,fg(h);h2Zg, is a nonnegative denite
Hermitian sequence in the sense of(1.20) if and only if there exists a nite non-
negative measurenon(p;p]such that
g(h) =
Z
p
p
e
ihw
n(dw);for all h2Z: (1.21)
This relation denesnuniquely.
Remark 1.18.By, g(h)that are auto-
covariance functions of second-order stationary processes. IffXt;t2Zgis station-
ary, thenn(ornxif we need to identify the process) is called thespectral measure
offXt;t2Zg. Thespectral distribution functiondened byFx(w) =nx(p;w]
is right-continuous, non-decreasing and bounded on[p;p]withFx(p) =0 and
Fx(p) =Var[Xt]. Ifnxadmits a densityfxwith respect to the Lebesgue measure
on(p;p], thenfxis referred to as thespectral density functionof the process
fXt;t2Zg. In this case,dFx(w) =fx(w)dw.
An important situation that simplies matters is the case where the autocovariance of
a stationary process is absolutely summable. In this case, the spectral measure admits
a density with respect to the Lebesgue measure.
Proposition 1.19 (Spectral Density).If the autocovariance functionfg(h);h2Zg,
of a stationary process satises
¥
å
h=¥
jg(h)j<¥; (1.22)
then it has the representation
g(h) =
Z
p
p
f(w)e
iwh
dw;h=0;1;2;:::; (1.23)
as the inverse transform of the spectral density, which has the representation
f(w) =
1
2p
¥
å
h=¥
g(h)e
iwh
0;pwp: (1.24)

37. 1.2. THE COVARIANCE WORLD 11
The proofs of
texts on time series such as2001),1991),
(1970),1981), or2011). The spectral density is
the analogue of the probability density function; the fact thatg(h)is non-negative
denite ensuresf(w)0 for allw2[p;p]. It follows immediately from (1.24)
thatf(w) =f(w). In addition, puttingh=0 in (1.23) yields
g(0) =Var(Xt) =
Z
p
p
f(w)dw;
which expresses the total variance as the integrated spectral density over all of the
frequencies.
Example 1.20 (Spectral density of white noise).As discussed in, if
fZt;t2Zgis white noise, WN(0;s
2
z), then its autocovariance function isgz(h) =
s
2
zdh(0). The absolute summability condition of
(1.24), its spectral density function is given by
fz(w) =
s
2
z
2p
;
forw2[p;p], which is a uniform density. Hence the process contains equal power
at all frequencies. In fact, the namewhite noisecomes from the analogy to white light,
which contains all frequencies in the color spectrum at the same level of intensity.3
Example 1.21 (Spectral measure of a harmonic process).Consider the complex-
valued harmonic processfXt;t2Zggiven by
Xt=
n
å
j=1
Aje
itwj
;p<w1<<wn<p; (1.25)
where theAjare uncorrelated complex-valued random variables such thatE[Aj] =0
andE

jAjj
2

=sj>0. It follows that the autocovariance function offXt;t2Zgis
given by
gx(h) =E[Xt+hX

t] =
n
å
j=1
s
2
je
ihwj
: (1.26)
Note that the total variance of the process is the sum of the variances of the individ-
ual components, Var[Xt] =gx(0) =å
n
j=1
s
2
j
. The autocovariance function does not
satisfy the absolute summability condition of, but we may write
gx(h) =
Z
p
p
s
2
je
ihl
dwj
(dl); (1.27)
wheredwj
denotes the Dirac mass at pointwj2(p;p). Consequently, the spectral
measure offXt;t2Zgis given by
nx=
n
å
j=1
s
2
jdwj
; (1.28)

38. 12 1. LINEAR MODELS
which is a sum of point masses with weightss
2
j
located at the frequencies of the
harmonic functions.
If the processfXt;t2Zgis real, thennmust be even and there must be conjugate
pairs in (1.25). For example, with n=2m, for 1jm, letwj+m=wj, write
Aj= (BjiCj)=2 andAj+m=A

j
, withfBjgandfCjgbeing uncorrelated random
variables with mean-zero and variances
2
j
. In this case,E

jAjj
2

=s
2
j
=2, so that the
spectral measure, (1.28), is now the sum of point masses with weights s
2
j
=2 atwj,
forj=1;:::;m. Note that
Xt=
n
å
j=1
Aje
itwj
=
m
å
j=1

BjiCj
2
e
itwj
+
Bj+iCj
2
e
itwj

=
m
å
j=1
Bj

e
itwj+e
itwj
2

iCj

e
itwje
itwj
2

=
m
å
j=1
Bjcos(wjt) +Cjsin(wjt):
BecauseBcos(wt)+Csin(wt)may be written asDcos(wt+f), whereB=Dcos(f)
andC=Dsin(f), we may also writeXt=å
m
j=1
Djcos(wjt+fj), wherefjis
called the (random)phaseof componentj. Also, in the real case, we havegx(h) =
å
m
j=1
s
2
j
cos(wjh). 3
In, we introduced the concept of a time invariant linear lter. In
general, a linear lter uses a set of specied coefcientsfaj;j2Zg, to transform an
input series,fXt;t2Zg, producing an output series,fYt;t2Zg, of the form
Yt=
¥
å
j=¥
ajXtj;
¥
å
j=¥
jajj<¥: (1.29)
The coefcientsfajgare called theimpulse response function, and the Fourier trans-
form
A(e
iw
) =
¥
å
j=¥
aje
iwj
; (1.30)
is called thefrequency response function.
The importance of the linear lter stems from its ability to enhance certain parts
of the spectrum of the input series. This fact is expressed in the following proposition;
see.
Proposition 1.22.If, in(1.29),fXt;t2Zghas spectral density fx, then
fy(w) =jA(e
iw
)j
2
fx(w); (1.31)
where fyis the spectral density offYt;t2Zgand the frequency response function
A(e
iw
)is dened in(1.30).
An important result that allows us to think of a stationary time series as being
approximately a random superposition of sines and cosines is the celebrated Cram´er
spectral representation theorem. The result is based on an orthogonal increment pro-
cess,Z(w), by which is meant a mean-zero, nite variance, continuous-time stochas-
tic process for which events occurring in non-overlapping intervals are uncorrelated.

39. 1.2. THE COVARIANCE WORLD 13
Theorem 1.23 (Cram´er).IffXt;t2Zgis a mean-zero stationary process, with
spectral measure as given in, then there exists a complex-valued or-
thogonal increment processfZ(w);w2(p;p]gsuch thatfXt;t2Zgcan be written
as the stochastic integral
Xt=
Z
p
p
e
itw
dZ(w);
where, forp<w1w2p,VarfZ(w2)Z(w1)g=n((w1;w2]):
In this text, we focus primarily on time domain approaches. More details on the
frequency domain approach to time series may be found in numerous texts such as
Brillinger2001),1991),1970),), or
Shumway and Stoffer2011).
1.2.3 Wold decomposition
The linear regression approach to modeling time series is generally implied by the
assumption that the dependence between adjacent values in time is best explained
in terms of a regression of the current values on the past values. This assumption is
partially justied, in theory, by the Wold decomposition. First, for completeness, we
state the following result for zero-mean, nite variance random variables.
Theorem 1.24 (L
2
projection theorem).Let X2L
2
and supposeMis a closed
subspace ofL
2
. Then X can be uniquely represented (a:e:) as
X=bX+Z
wherebX2 Mand Z is orthogonal toM; i.e.,E[ZW]=0 for all W inM. Fur-
thermore, the pointbX is the closest to X in the sense that, for any W2 M,
E[XW]
2
E[XbX]
2
, where equality holds if and only if W=bX(a:e:).
We call the mappingPMX=bX, forX2L
2
, the orthogonal projection mapping of
XontoM. Recall that the closed span of setfX1;:::;Xngof elements in L
2
is dened
to be the set of all linear combinationsW=a1X1++anXn, wherea1;:::;anare
scalars. This subspace of L
2
is denoted byM=spfX1;:::;Xng.
For a stationary, zero-mean processfXt;t2Zg, we dene
M
x
n=spfXt;¥<tng;withM
x
¥=
¥
\
n=¥
M
x
n;
and
s
2
x=E

Xn+1PM
x
n
Xn+1

2
;
wherePM
x
n
()denotes orthogonal projection onto the spaceM
x
n. We say thatfXt;t2
Zgis a deterministic process if and only ifs
2
x=0. That is, a deterministic process
is one in which its future is perfectly predictable from its past; e.g.,Xt=Acos(pt),
whereA2L
2
. We are now ready to present the decomposition.

40. 14 1. LINEAR MODELS
Theorem 1.25 (The Wold decomposition).Under the conditions and notation
previously mentioned, ifs
2
x>0, then Xtcan be expressed as
Xt=
¥
å
j=0
yjZtj+Vt
where
(i)å
¥
j=0
y
2
j
<¥(y0=1)
(ii)fZt;t2Zg WN(0;s
2
z)
(iii) t2 M
x
t
(iv)Cov(Zs;Vt) =0for all s;t=0;1;2;:::
(v) t2 M
x
¥
(vi)fVt;t2Zgis deterministic.
The proof of the decomposition follows from
unique sequences
Zt=XtP
M
x
t1
Xt;yj=s
2
zE[XtZtj];Vt=Xt
¥
å
j=0
yjZtj:
Details may be found in1991). Although every stationary
process can be represented by the Wold decomposition, it does not mean that the
decomposition is the best way to describe the process. In addition, there may be some
dependence structure among thefZtg; we are only guaranteed that the sequence is
an uncorrelated sequence. The theorem, in its generality, falls short in that we would
prefer the noise process,fZtg, to be i.i.d. noise.
By imposing extra structure,conditional expectationcan be dened as a pro-
jection mapping for random variables in L
2
with the equivalence relation that, for
X;Y2L
2
,X=YifP(X=Y) =1. In particular, forY2L
2
, ifMis a closed sub-
space of L
2
containing 1, the conditional expectation ofYgivenMis dened to be
the projection ofYontoM, namely,EM[Y]E[YjM] =PMY:This means that con-
ditional expectation,EM, must satisfy the orthogonality principle of the Projection
Theorem,. If we let M(X)denote the closed subspace of all random
variables in L
2
that can be written as a measurable function ofX, then we may dene,
forX;Y2L
2
, theconditional expectation of Y given XasE[YjX]:=E[YjM(X)].
This idea may be generalized in an obvious way to dene the conditional expectation
ofYgivenXXX= (X1;:::;Xn); that isE[YjXXX]:=E[YjM(XXX)]. Of particular interest to
us is the following result, which states that, in the Gaussian case, conditional expec-
tation and linear prediction are equivalent.
Theorem 1.26.If(Y;X1;:::;Xn)is multivariate normal, then
E

Y


X1;:::;Xn

=P
spf1;X1;:::;XngY:
Proof.It follows from YgivenXXX=
(X1;:::;Xn)is the unique elementE
M(XXX)Ythat satises the orthogonality principle,
E

YE
M(XXX)Y

W

=0 for allW2 M(XXX):

41. 1.3. LINEAR PROCESSES 15
We must show thatbY=P
spf1;X1;:::;XngYis that element. In fact, by the Projection
Theorem (Theorem 1.24), bYsatises
E
h
(YbY)Xi
i
=0 fori=0;1;:::;n;
where we have setX0=1. ButE[(YbY)Xi] =Cov(YbY;Xi) =0, implying that
YbYandfX1;:::;Xngare independent because the vector(YbY;X1;:::;Xn)
0
is
multivariate normal. Thus, ifW2 M(XXX), thenWandYbYare independent and,
hence,Ef(YbY)Wg=E(YbY)E(W) =0, recalling thatE(YbY) =0.
In the Gaussian case, conditional expectation has a simple explicit form. Let
YYY= (Y1;:::;Ym)
0
,XXX= (X1;:::;Xn)
0
, and suppose the(m+n)1 vector(YYY
0
;XXX
0
)
0
is normal:
YYY
XXX

N

mmm
y
mmm
x

;

SyySyx
SxySxx

;
thenYYYjXXXis normal with
mmm
yjx
=mmm
y
+SyxS
1
xx(XXXmmm
x
) (1.32)
S
yjx=SyySyxS
1
xxSxy; (1.33)
whereSxxis assumed to be nonsingular.
1.3 Linear processes
1.3.1 What are linear Gaussian processes?
The concept of second-order stationarity forms the basis for much of the analysis
performed with linear Gaussian time series. Most models used in this situation are
special cases of the linear process.
Denition 1.27 (Linear process).Alinear process,fXt;t2Zg, is dened to be a
linear combination of white noise variates ZtWN(0;s
2
z), and is given by
Xt=m+
¥
å
j=¥
yjZtj;
¥
å
j=¥
y
2
j<¥: (1.34)
The stronger condition of absolute summability, i.e.,å
¥
j=¥jyjj<¥, is often
required and, when needed, we will make this distinction explicit. For example, the
time invariant linear lter (see) is dened in such a way. Another
frequent additional condition is thatZtiid(0;s
2
z); again, we will make this distinc-
tion when necessary. For a linear process, note thatXt2L
2
; also, direct calculation
yields the autocovariance function,
gx(h) =s
2
z
¥
å
j=¥
yjy
j+jhj; (1.35)
forh2Z.

42. 16 1. LINEAR MODELS
Denition 1.28 (Gaussian process).A process,fXt;t2Zg, is said to be aGaus-
sian processif the n-dimensional vectors XXX= (Xt1
;Xt2
;:::;Xtn
)
0
, for every collection
of distinct time points t1;t2;:::;tn, and every positive integer n, have a nonsingular
multivariate normal distribution.
Dening then1 mean vectorE(XXX)mmm= (mt1
;mt2
;:::;mtn
)
0
and thenn
variance-covariance matrix as Cov(XXX)G=fg(ti;tj);i;j=1;:::;ng, which is as-
sumed to be positive denite, the multivariate normal density function is
g(xxx;mmm;G) = (2p)
n=2
jGj
1=2
exp


1
2
(xxxmmm)
0
G
1
(xxxmmm)

; (1.36)
wherej jdenotes the determinant andxxx2R
n
. This distribution forms the basis for
solving problems involving statistical inference for linear Gaussian time series. If a
Gaussian time series is weakly stationary, thenmt=mandg(ti;tj) =g(jtitjj), so
that the vectormmmand the matrixGare independent of time,t. These facts imply that
all the nite distributions, (1.36), of the series fXtgdepend only on time lag and not
on the actual times, and hence the series must be strictly stationary.
If, in, we assume that Ztiid N(0;s
2
z), then we are in the friendly
world of stationary linear Gaussian processes. Many real processes can be approxi-
mated (i.e., modeled) under these assumptions, sometimes after coercion via simple
transformations such as differencing or logging, and this forms the basis of classi-
cal, or second-order, time series analysis. However, many interesting processes are
patently not linear or Gaussian, and that is the focus of this text. We note that linear
processes need not be Gaussian, but by, (non-deterministic) stationary
Gaussian processes are linear.
1.3.2 ARMA models
We briey describe autoregressive-moving average (ARMA) models that were pop-
ularized by the work of1970). There are many other texts that
present an exhaustive treatment of these and associated models, e.g.,
Davis1991),1996), or2011).
Denition 1.29 (ARMA model).A time seriesfXt;t2ZgisARMA(p;q)if it is
stationary and
Xt=f1Xt1++fpXtp+Zt+q1Zt1++qqZtq; (1.37)
withfp6=0,qq6=0, and ZtWN(0;s
2
z>0). The parameters p and q are called
the autoregressive and the moving average orders, respectively. If Xthas a nonzero
meanm, we setf0=m(1f1 fp)and write the model as
Xt=f0+f1Xt1++fpXtp+Zt+q1Zt1++qqZtq: (1.38)
Although it is not necessary for the denition, it is typically assumed for the sake
of inference and prediction thatZtis Gaussian white noise. Whenq=0, the model
is called an autoregressive model of orderp, AR(p), and whenp=0, the model is

43. 1.3. LINEAR PROCESSES 17
called a moving average model of orderq, MA(q). It is sometimes advantageous to
write the ARMA(p;q) model in concise form as
f(B)Xt=q(B)Zt; (1.39)
whereBis the backshift operator
BXt=Xt1;
withB
k
Xt=B
k1
(BXt) =Xtk, and where theautoregressive operatoris dened to
be
f(B) =1f1Bf2B
2
fpB
p
; (1.40)
and themoving average operatoris
q(B) =1+q1B+q2B
2
++qqB
q
: (1.41)
Denition 1.30 (Causal/nonanticipative and invertible).A linear process,fXt;t2
Zg, is said to becausalornonanticipativeif it can be written as a one-sided linear
process,
Xt=
¥
å
j=0
yjZtj=y(B)Zt; (1.42)
wherey(B) =å
¥
j=0
yjB
j
, andå
¥
j=0
jyjj<¥; we sety0=1:A linear process is said
to beinvertibleif it can be written as
p(B)Xt=
¥
å
j=0
pjXtj=Zt; (1.43)
wherep(B) =å
¥
j=0
pjB
j
, andå
¥
j=0
jpjj<¥; we setp0=1:
More generally, a processfXt;t2Zgis said to becausalornonanticipativeif
Xt=G(Zs;st)where G is a measurable function such that Xtis a properly dened
random variable. Likewise, if Zt=G(Xs;st), the processfXt;t2Zgis said to be
invertible.
Causality (nonanticipativity) assures that the process depends only on the present
and past innovationsZt, and not on future errors. Invertibility is a similar useful prop-
erty of the noise process; for example, see. We now state the conditions
for which an ARMA model is both causal and invertible.
Proposition 1.31 (Causality and invertibility of ARMA).Let z2Cand assume
that there is no z for whichf(z) =q(z) =0, wheref()andq()are dened in (1.40)
and (1.41), respectively. An ARMA(p;q) model is causal if and only iff(z)6=0for
jzj 1. The coefcients of the linear process given in (1.42) can be determined by
solving
y(z) =
¥
å
j=0
yjz
j
=
q(z)
f(z)
;jzj 1:

44. 18 1. LINEAR MODELS
An ARMA(p;q) model is invertible if and only ifq(z)6=0forjzj 1. The coefcients
pjofp(B)given in (1.43) can be determined by solving
p(z) =
¥
å
j=0
pjz
j
=
f(z)
q(z)
;jzj 1:
Example 1.32 (ACF and PACF of ARMA).In view of causality, the autocovari-
ance function of an ARMA(p;q) process is given by (1.35). Consequently, the ACF
is given by
r(h) =
å
¥
j=0
yjy
j+jhj
å
¥
j=0
y
2
j
: (1.44)
If the process is pure MA(q), theny0=1,yj=qjforj=1;:::;q, andyj=0
forj>q. Thus, for a pure MA process,r(h) =0 forjhj>q, in which case the
ACF can be used to identify the orderq. For an AR(1),f(z) =1fz, so that using
Proposition 1.31, it is seen that yj=f
j
. In this case,r(h) =f
jhj
, so that the ACF
does not zero-out at any lagh. In general, for an AR(p) or ARMA(p;q), the ACF is
not helpful in determining the orders because it does not zero-out as in the MA case.
A useful measure for determining the order of an AR(p) model ispartial au-
tocorrelation function(PACF), which extends the idea of partial correlation to time
series. Recall that if(X;Y;ZZZ)is multivariate normal, then Cor(X;Y

ZZZ)is the corre-
lation coefcient betweenXandYin the bivariate normal conditional distribution of
XandYgivenZZZ= (Z1;:::;Zk). This value is seen as the correlation betweenXand
Ywith the effect ofZZZremoved (or partialled out). The denition can be extended to
non-normal variables by dening Cor(X;Y


ZZZ)to be Cor(XPzX;YPzY), where
Pzdenotes projection (or regression) ontosp(1;Z1;:::;Zk). This denition is equiv-
alent to the original denition in the Gaussian case. For a stationary time series,Xt,
the PACF, denotedfhh, forh=1;2;:::;is dened to be
f11=Cor(Xt+1;Xt) =r(1) (1.45)
and
fhh=Cor(Xt+hPh1Xt+h;XtPh1Xt);h2; (1.46)
wherePh1denotes projection ontospf1;Xt+1;Xt+2;:::;Xt+h1g. Thus, the PACF is
seen as the correlation betweenXt+handXtwith the effect of everything in the middle
being partialled out. For example, for a zero-mean AR(1) process,Xt=fXt1+Zt,
we havef11=r(1) =fand
f22=Cor(Xt+2P1Xt+2;XtP1Xt) =Cor(Xt+2fXt+1;XtP1Xt)
=Cor(Zt+2;XtP1Xt) =0
becauseXtP1Xtis a function ofZt;Zt1;:::, which are uncorrelated withZt+2.
Similar calculations will showfhh=0 forh>1. Similarly, for a general AR(p), we
havefhh=0 forh>p, which can be used to identify the order of the AR.3

45. 1.3. LINEAR PROCESSES 19
Example 1.33 (Spectrum of ARMA).Using the results and notation of
tion 1.22, the spectral density of an ARMA( p;q) process is
given by
fx(w) =fz(w)jy(e
iw
)j
2
=
s
2
z
2p
jq(e
iw
)j
2
jf(e
iw
)j
2
; (1.47)
recalling from fz(w) =s
2
z=2p. 3
1.3.3 Prediction
In prediction or forecasting, the goal is to predict future values of a time series,
Xn+m,m=1;2;:::, based on the data collected to the present,X1:n=fX1;:::;Xng.
LetFn=s(X1;:::;Xn), then the minimum mean square error predictor ofXn+mis
X
n+mjn:=E

Xn+m

Fn

(1.48)
because the conditional expectation minimizes the mean square error
E[Xn+mg(X1:n)]
2
; (1.49)
whereg:R
n
!Ris a function of the observations.
Except in the Gaussian case, it is typically difcult to perform optimal prediction
because all the nite dimensional distributions must be known to compute (1.48).
Typically, one makes a compromise and works with best linear prediction; that is,
predictors of the form
X
n+mjn=f
(m)
n0
+
n
å
j=1
f
(m)
n j
Xn+1j; (1.50)
wheref
(m)
n0
;f
(m)
n1
;:::;f
(m)
nnare real numbers. Linear predictors of the form (1.50) that
minimize the mean square prediction error (1.49) are called best linear predictors
(BLPs). Linear prediction depends only on the second-order moments of the pro-
cess, which are easy to estimate from the data in the stationary case. If, in addition,
the process is Gaussian, minimum mean square error predictors and best linear pre-
dictors are the same. We note that in the stationary case we can eliminatef
(m)
n0
by
centering the process; i.e., in (1.50), we replace f
(m)
n0
bymandXtbyXtm; conse-
quently, we dropf
(m)
n0
from the discussion. The following property, which is based
on, is a key result; e.g., see2011).
Proposition 1.34 (Best linear prediction for stationary process).Given observa-
tions X1;:::;Xn, the (a.e. unique) best linear predictor, X
n+mjn=å
n
j=1
f
(m)
n j
Xn+1j;
of Xn+m, for m1, is found by solving
E

(Xn+mX
n+mjn)Xt

=0;t=1;:::;n; (1.51)
forf
(m)
n1
;:::;f
(m)
nn. The prediction equations (1.51) can be written in matrix notation
as
Gnf
(m)
n=g
(m)
n; (1.52)

46. 20 1. LINEAR MODELS
wheref
(m)
n= (f
(m)
n1
;:::;f
(m)
nn)
0
andg
(m)
n= (g(m);:::;g(m+n1))
0
are n1vec-
tors, andGn=fg(ij)g
n
i;j=1
is an nn matrix, IfGnis non-singular, then thefs are
unique and may be obtained as
f
(m)
n=G
1
ng
(m)
n: (1.53)
In addition, the m-step-ahead mean square prediction error can be computed as
P
n+mjn:=E[Xn+mX
n+mjn]
2
=g(0)g
(m)
0
nG
1
ng
(m)
n: (1.54)
Moreover,f
(1)
hh
as dened here, is equivalent to the PACF,fhh, dened in (1.45)–
(1.46).
The Durbin-Levinson algorithm,1947) and1960), can be
used to obtain the components off
(1)
nrecursively,n=1;2;:::, without having to
invertGn. Most texts on linear time series discuss the algorithm. We also note that
Proposition 1.34 fhh, is a function ofg(h)alone, and conse-
quently it may be estimated easily from the data via (1.15).
Example 1.35 (BLP versus minimum mean square prediction).SupposeXand
Zare independent standard normal random variables and let
Y=X
2
+Z:
We wish to predictYbased on the dataX. The minimum mean square predictor is
given byE[YjX] =X
2
with mean square prediction error (MSPE)
E[YE(YjX)]
2
=E[YX
2
]
2
=EZ
2
=1:
Now, letg(X) =a+bXbe the BLP. Then using, g(X)satises
E[Yg(X)] =0 andE[(Yg(X))X] =0;
or
E[Y] =E[a+bX]andE[XY] =E[(a+bX)X]:
From the rst equation we havea+bE[X] =E[Y], butE[X] =0 andE[Y] =1,
soa=1. From the second equation we haveaE[X] +bE[X]
2
=E[XY], orb=
E

X(X
2
+Z)

=E[X
3
] +E[XZ] =0+0. Consequently, the BLP isg(X) =1, in-
dependent of the dataX, and thus the MSPE is
E[Y1]
2
=E

Y
2

1=E[X]
4
+E[Z]
2
1=3+11=3;
noting that the fourth moment of a standard normal is 3.
We see that using linear prediction in a nonlinear setting can give strange results;
here, the predictor does not even rely on the data and the BLP has three times the
error of the optimal predictor (conditional expectation). Three times the error may
not seem that large at rst, but we forgot to mention at the start of this example that
the units of the problem are in trillions of dollars. 3

47. 1.3. LINEAR PROCESSES 21
As previously mentioned, BLP and minimum mean square prediction are the
equivalent in the Gaussian case. For example, given data vectorXXXn= (Xn;:::;X1)
0
,
suppose we are interested in the one-step-ahead predictor,X
n+1jn, and the corre-
sponding MSPE,P
n+1jn. Then, using standard multivariate normal distribution the-
ory, we may write
X
n+1jn=m+ggg
0
n
G
1
n(XXXnmmm
n
) (1.55)
P
n+1jn=g(0)ggg
0
n
G
1
nggg
n
; (1.56)
wherem=EXt,mmm
n
=EXXXn= (m;:::;m)
0
andggg
n
=Cov(Xn+1;XXXn) = (g(1);:::;g(n))
0
aren1 vectors, andGn=Cov(XXXn) =fg(ij)g
n
i;j=1
is annnmatrix.
For ARMA models, many simplications are available. For example, ifXtis
AR(1), then, forn1,X
n+1jn=f0+f1XnandP
n+1jn=E[Xn+1(f0+f1Xn)]
2
=
E

Z
2
t

=s
2
z. In addition,X
1j0=E[X1] =m=f0=(1f1)andP
1j0=Var[X1] =
s
2
z=(1f1)
2
; see. It should be evident that prediction for pure AR(p)
models, whennp, parallels the AR(1) case; i.e.,
X
n+1jn=f0+f1Xn+11+f2Xn+12++fpXn+1p; (1.57)
andP
n+1jn=s
2
z. Forn<p, the prediction equations (1.55) – (1.56) can be used.
For general ARMA(p;q), one can simplify matters by assuming that a complete
history is available; i.e., the data arefXn;:::;X1;X0;X1;:::g. Using invertibility, we
can writeXn+1=å
¥
j=1
pjXn+1j+Zt. Consequently,
X
n+1jn=
¥
å
j=1
pjXn+1j: (1.58)
Of course, only the dataX1;:::;Xnare available, so (1.58) is truncated to X
n+1jn
å
n
j=1
pjXn+1j, the idea being that ifnis sufciently large, there is negligible dif-
ference between (1.58) and the truncated version. From (1.58), it is also evident that
P
n+1jn=E

(Xn+1X
n+1jn)
2

=E

Z
2
n+1

=s
2
z;and this approximation is used in
the truncated case. Forecasting ARMA processesmsteps ahead uses similar approx-
imations, see, e.g.,2011, Chapter 3).
1.3.4 Estimation
Time domain
Estimation for pure AR(p) models is fairly easy because they are essentially linear
regression models. For example, for the AR(1) model,Xt=f0+f1Xt1+Zt, given
a realizationx1;:::;xn, one can perform ordinary least squares (“Y=b0+b1x+e”)
on then1 data pairsf(y;x):(x2;x1);(x3;x2);:::;(xn;xn1)g. The approach for an
AR(p) is similar, and the technique is efcient. An alternate method that is also
efcient is Yule–Walker estimation, which is method of moments estimation (MME).
Recall that in MME, we equate population moments to sample moments and then
solve for the parameters in terms of the sample moments. BecauseE[Xt] =m, the

48. 22 1. LINEAR MODELS
MME ofmis the sample average,X; thus to ease the notation, we will assumem=0.
For an AR(p) process,
Xt=f1Xt1++fpXtp+Zt;
multiply through byXth, forh0 and take expectation to obtain the followingp+1
Yule–Walker equations (see),
g(h) =f1g(h1) ++fpg(hp);h=1;2;:::;p; (1.59)
s
2
z=g(0)f1g(1) fpg(p): (1.60)
Next, replaceg(h)withbg(h)(see (1.15)) and solve for fs ands
2
z.
Maximum likelihood estimation for normal ARMA(p;q)models proceeds as fol-
lows. Letb= (m;f1;:::;fp;q1;:::;qq)
0
be the(p+q+1)-dimensional vector of the
model parameters. The likelihood can be written as
L(b;s
2
z;X1:n) =
n
Õ
t=1
p
b;s
2
z(Xt

Xt1;:::;X1);
where p
b;s
2
z( j )is the N(X
tjt1;P
tjt1)distribution. For the ARMA model, the noise
variance,s
2
z, may be factored out and we can writeP
tjt1=s
2
zrt(b), wherert()
depends only onb. This fact can be seen from (1.56) by factoring out s
2
zfrom the
g(h)terms; recall (1.35). The likelihood of the data can now be written as
L(b;s
2
z;X1:n) = (2ps
2
z)
n=2
[r1(b)r2(b)rn(b)]
1=2
exp


S(b)
2s
2
z

;(1.61)
where
S(b) =
n
å
t=1
(
XtX
tjt1(b)

2
rt(b)
)
: (1.62)
Note thatX
tjt1is also a function ofbalone, and we make that fact explicit in (1.61)
– (1.62). Given a realizationx1;:::;xn, and values forbands
2
z, the likelihood may
be evaluated using the prediction techniques of the previous subsection.
Maximum likelihood estimation would now proceed by maximizing (1.61) with
respect tobands
2
z. Note that
bs
2
z=n
1
S(
b
b); (1.63)
where
b
bis the value ofbthat minimizes the concentrated or prole likelihood,
l(b) =ln

n
1
S(b)

+n
1
n
å
t=1
lnrt(b); (1.64)
wherel(b)µ2lnL(b;bs
2
z):
Becausel(b)is a complicated function ofp+q+1 parameters, optimization

49. 1.3. LINEAR PROCESSES 23
routines are employed. Numerical methods for accomplishing maximum likelihood
estimation are discussed in. When tting ARMA models to data, one often
considers multiple models. In such a cases, we often rely on the theory of parsi-
mony to choose the best model. The choice is typically aided by the use of various
information theoretic criteria that take the general form
Dk;n=2lnLk+Ck;n; (1.65)
where Lkis the value of the maximized likelihood,kis the number of regression
parameters in the model,nis the sample size andCk;nis a penalty for adding
parameters. The most used are (i) Akaike's Information Criterion (AIC), wherein
Ck;n=2k,1973,); (ii) Corrected AIC (AICc), wherein Ck;n=
n+k
nk2
,
Hurvich and Tsai1993); (iii) Bayesian (or Schwarz's) Information Criterion (BIC),
whereinCk;n=kln(n),1978); and (iv) Hannan-Quinn (HQ), wherein
Ck;n=2klnln(n),1979). Most texts on time series analysis pro-
vide further discussion of selection criteria; see1998) for a
general reference.
Frequency domain
There are two basic methods for estimating a spectral density; parametric estimation
and nonparametric estimation. In, we exhibited the spectrum of an
ARMA process and we might consider basing a spectral estimator on this function,
substituting the parameter estimates from an ARMA(p;q) t on the data into the
formula for the spectral densityfx(w)given in (1.47). Such an estimator is called
a parametric spectral estimator. For convenience, a parametric spectral estimator is
obtained by tting an AR(p) to the data, where the orderpis determined by one of
the model selection criteria. Parametric autoregressive spectral estimators will often
have superior resolution in problems when several closely spaced narrow spectral
peaks are present and are preferred by engineers for a broad variety of problems;
e.g., see1988). The development of autoregressive spectral estimators has been
summarized by1983).
Ifbf1;bf2;:::;bfpandbs
2
zare the estimates from an AR(p) t to observations
X1;:::;Xn, then a parametric spectral estimate offx(w)is attained by substituting
these estimates into (1.47), that is,
bfx(w) =
bs
2
z
2p

bf

e
iw


2
; (1.66)
where
bf(z) =1bf1zbf2z
2
bfpz
p
: (1.67)
An interesting fact about rational spectra of the form (1.47) is that any spectral den-
sity can be approximated, arbitrarily close, by the spectrum of an AR process; e.g.,
see1991) or1996). In a sense, we can say that the
spectral densities of AR processes are dense in the space of continuous spectral den-
sities.

50. 24 1. LINEAR MODELS
Nonparametric spectral estimation is a little more involved and for full details,
we refer the reader to other texts such as1991) or
and Stoffer2011). The basic building block is the discrete Fourier transform(DFT).
Given dataX1;:::;Xn, the DFT is dened to be
d(wj) = (2pn)
1=2
n
å
t=1
Xte
iwjt
(1.68)
forj=0;1;:::;n1, where the frequencieswj=2pj=nare called theFourieror
fundamental frequencies. Theperiodogramis then dened as the squared modulus
of the DFT,
I(wj) =

d(wj)


2
(1.69)
forj=0;1;2;:::;n1. ReplacingXtbyXtXin (1.68), it is easily shown (e.g.,
Shumway and Stoffer,) that
I(wj) =
1
2p
n1
å
h=(n1)
bg(h)e
iwjh
;
forwj6=0;
1
2
, which, in view of (1.24), shows that the periodogram may be consid-
ered the “sample” spectral density. One can extend the peridogram to allw2[p;p]
by deningI(w) =I(wj)forjwjwj 1=2nandI(w) =I(w). Although
E[I(w)]!f(w)
asn!¥, it can be shown that under mild conditions, Var[I(w)]!f
2
(w), so that the
periodogram is not a consistent estimator of the spectral densityf(w). This problem
is overcome by local smoothing of the periodogram in a neighborhood of a frequency
of interest,fwj
mn
n
wwj+
mn
n
g, say
bf(w) =
mn
å
k=mn
hkI(wj+k=n); (1.70)
wheremn!¥butmn=n!0 asn!¥, and the weightshk>0 satisfy (n!¥)
mn
å
k=mn
hk=1 and
mn
å
k=mn
h
2
k
!0:
In essence, one may think of the periodogram as a histogram, and (1.70) as smoothing
the histogram to obtain an estimate of the smooth densityf(w). A simple average
corresponds to the case wherehq=1=(2m+1)forq=m;:::;0;:::;m. The number
mis chosen to obtain a desired degree of smoothness. Larger values ofmlead to
smoother estimates, but one has to be careful not to smooth away signicant peaks
(this is the bias-variance tradeoff problem). Experience and trial-and-error can be
used to select good values ofmand the set of weightsfhqg. Another consideration is
that of tapering the data prior to a spectral analysis; i.e. rather than work with the data

51. 1.4. THE MULTIVARIATE CASES 25
Xtdirectly, one can improve the estimation of spectra by working with tapered data,
sayYt=atXt, where tapersfatggenerally have a shape that enhances the center of the
data relative to the extremities, such as a cosine bell,at=:5[1+cos(2pt
0
=n)]where
t
0
=t(n+1)=2, favored by1959). Another related approach
is window spectral estimation. Specically, consider a window functionH(a),¥<
a<¥, that is real-valued, even, of bounded variation, with
R
¥
¥
H(a)da=1, and
R
¥
¥
jH(a)jda<¥. The window spectral estimator is
bf(w) =n
1
n1
å
q=1
Hn(wq=n)I(q=n); (1.71)
whereHn(a) =B
1
nå
¥
j=¥H(B
1
n[a+j])andBnis a bounded sequence of non-
negative scale parameters such thatBn!0 andnBn!¥asn!¥. Estimation
of the spectral density requires special attention to the issues of leakage and of the
variance-bias tradeoff typically associated with the estimation of density functions.
Readers who are unfamiliar with this material may consult one of the many texts
on the spectral domain analysis of time series; e.g.,2001),
(2004), or2011, Chapter 4).
1.4 The multivariate cases
1.4.1 Time domain
In this situation, we are interested in modeling and forecastingk1 vector-valued
time seriesXt= (Xt1;:::;Xtk)
0
, fort=0;1;2;:::. Unfortunately, extending uni-
variate ARMA models to the multivariate case is not so simple. The multivariate
autoregressive model, however, is a straight-forward extension of the univariate AR
model.
Ak1 vector-valued time seriesXt, fort=0;1;2;:::, is said to be
VARMA(p;q)ifXtis stationary and
Xt=F0+F1Xt1++FpXtp+Zt+Q1Zt1++QqZtq; (1.72)
withFp6=0,Qq6=0, and thevector white noiseprocessZt, typically taken to be mul-
tivariate normal with mean-zero and variance-covariance matrixSz>0 (that is,Szis
positive denite). The coefcient matricesFjforj=1;:::;pandQjforj=1;:::;q
arekkmatrices. Ifq=0, the model is a pure VAR(p) model, and ifp=0, the
model is a pure VMA(q) model. IfXthas meanmthenF0= (IF1 Fp)m.
As in the univariate case, a number of conditions must be placed on the multivariate
ARMA model to ensure the model is unique and has desirable properties such as
causality. The special form assumed for the constant componentF0can be general-
ized to include a xedr1 vector of inputs,Ut. That is, we could have proposed the
vector ARMAX model,
Xt=¡Ut+
p
å
j=1
FjXtj+
q
å
k=1
QkZtk+Zt; (1.73)

52. 26 1. LINEAR MODELS
where¡is akrparameter matrix.
In the multivariate case, theautoregressive operatoris
F(B) =IF1B FpB
p
; (1.74)
and themoving average operatoris
Q(B) =I+Q1B++QqB
q
; (1.75)
The zero-mean VARMA(p;q) model is then written in the concise form as
F(B)Xt=Q(B)Zt: (1.76)
The model is said to becausalif the roots ofF(z)lie outside the unit circle, i.e.,
detfF(z)g 6=0 for any valuezsuch thatjzj 1. In this case, we can write
Xt=Y(B)Zt;
whereY(B) =å
¥
j=0
YjB
j
,Y0=I, andå
¥
j=0
jjYjjj<¥:The model is said to be
invertibleif the roots ofQ(z)lie outside the unit circle. Then, we can write
Zt=P(B)Xt;
whereP(B) =å
¥
j=0
PjB
j
,P0=I, andå
¥
j=0
jjPjjj<¥:Analogous to the univariate
case, we can determine the matricesYjby solvingY(z) =F(z)
1
Q(z);forjzj 1,
and the matricesPjby solvingP(z) =Q(z)
1
F(z);forjzj 1:
For a causal model, we can writeXt=Y(B)Ztso the general autocovariance
structure of an ARMA(p;q) model is
G(h) =Cov(Xt+h;Xt) =
¥
å
j=0
Yj+hSzY
0
j: (1.77)
andG(h) =G(h)
0
. For pure MA(q) processes, (1.77) becomes
G(h) =
qh
å
j=0
Qj+hSzQ
0
j; (1.78)
whereQ0=I. Of course, (1.78) implies G(h) =0 forh>q.
For pure VAR(p) models, the autocovariance structure leads to the multivariate
version of theYule–Walker equations
G(h) =
p
å
j=1
FjG(hj);h=1;2;:::; (1.79)
G(0) =
p
å
j=1
FjG(j) +Sz: (1.80)
whereG(h) =Cov(Xt+h;Xt)is akkmatrix. Analogous to the univariate case,

53. 1.4. THE MULTIVARIATE CASES 27
these equations can be used to obtain method of moment estimators of the model
parameters by replacingG(h)with the corresponding sample moments, and solving
for theFs andSz. For moment estimation of the autocovariance matrix, we setX=
n
1
å
n
t=1
Xt, as an estimate ofm=EXt,
bG(h) =n
1
nh
å
t=1
(Xt+hX)(XtX)
0
;h=0;1;2;::;n1; (1.81)
andbG(h) =bG(h)
0
.
As previously mentioned, there are many problems with the general VARMA
model that have to do with parameter uniqueness and estimation. For further details,
the reader is referred to ¨utkepohl2005),2003), or
(2011). These problems may be avoided by using only pure models, and typically that
means relying on VAR(p) models. In this case, estimation is fairly straight-forward
via the Yule–Walker equations, or by realizing that conditional on a few stating val-
ues,X1;:::;Xp, the model is multivariate linear regression. Maximum likelihood es-
timation in the multivariate setting can also be performed numerically as in the scalar
case, but in this case,Szcannot be factored out of the likelihood.
1.4.2 Frequency domain
As in, it is convenient to work with complex-valued time series. Ap1
complex-valued time series can be represented asXt=X1tiX2t;whereX1tis the real
part andX2tis the imaginary part ofXt. The process is said to be stationary ifE(Xt)
andE(Xt+hX

t)exist and are independent of timet, where

denotes conjugation and
transposition. Theppautocovariance function,
G(h) =E(Xt+hX

t)E(Xt+h)E(X

t);
ofXtsatises conditions similar to those of the real-valued case. WritingG(h) =
fgi j(h)g, fori;j=1;:::;p, we have (i)gii(0)0 is real, (ii)jgi j(h)j
2
gii(0)gj j(0)
for all integersh, and (iii)G(h)is a non-negative denite function. The spectral
theory of complex-valued vector time series is analogous to the real-valued case. For
example, ifåhjjG(h)jj<¥, the spectral density matrix of the complex seriesXtis
given by
f(w) =
1
2p
¥
å
h=¥
G(h)e
ihw
:
The off-diagonal elements off(w), sayfi j(w), fori6=j=1;:::;pare called
cross-spectra. Typically one is interested insquared-coherence, which is a frequency
based measure of squared correlation, dened as
r
2
i j(w) =
jfi j(w)j
2
fii(w)fj j(w)
: (1.82)
As with squared correlation, 0r
2
i j
(w)1, with values close to one indicating a
strong linear relationship betweenXitandXjtat frequencyw.

54. 28 1. LINEAR MODELS
Estimation can be achieved, as in the univariate case, by parametric or nonpara-
metric methods. If a VAR(p) model is assumed for the process, then the spectral
density is
f(w) =
1
2p
F
1
(e
iw
)SzF
01
(e
iw
);
whereF(z)is given in (1.74). Once the VAR parameters are estimated, one simply
substitutes the parameters for their estimates in the above. In the nonparametric case,
the vector DFT is calculated,
d(wj) = (2pn)
1=2
n
å
t=1
Xte
iwjt
;
in which case the periodogram is appmatrix,I(wj) =d(wj)d(wj)

. Consistency
and non-negative deniteness are still concerns, so a smoothed estimator is used,
bf(w) =
mn
å
k=mn
hkI(wj+k=n):
Here, thehkare scalars and satisfy the conditions given in the univariate case.
1.5 Numerical examples
In this section we provide brief numerical examples on tting ARMA models and
performing spectral analysis on anRdata set. In both examples we use theRdata set
sunspot.year, which are the annual sunspot numbers for the years 1700–1988. Be-
cause there are extreme values, we rst transform the data by taking the square root.
The transformed data are displayed in Rcode displayed in the
rst two lines of. It has been noted by many authors that the sunspot
series exhibits nonlinear and non-normal behavior. We will discuss this problem in
more detail throughout the text, but as a quick check, note that in, the
sunspot numbers tend to rise quickly to a peak and then decline slowly to a trough
("&); thus the series resembles a reverse, or inverse, saw tooth wave. Thus, if put
in reverse-time order, the data will rise slowly and decline quickly. A linear Gaus-
sian process cannot have this behavior because the distribution offX1;:::;Xngmust
be the same asfXn;:::;X1g; i.e., both have the same mean vector,m, and variance-
covariance matrix,G, as dened in (1.36). Nevertheless, for the sake of demonstra-
tion, we will apply linear models to the series.
Example 1.36 (Sunspots – ARMA).Continuing with the sunspot data, plotting the
sample ACF and PACF (not shown) suggest tting an autoregression of at least order
p=2. We then used theRcommandar()to search for the best autoregressive
model based on AIC when the parameters are estimated by Yule-Walker. This method
suggests an AR(9) model, which is then t using maximum likelihood. The nal
results are:

55. Another Random Document on
Scribd Without Any Related Topics

56. Johannes perceived that "converted" meant "very good." According
to Daatje, the natural man was not good, and it was necessary for
every one to be converted before he was fit for anything. For a long
time before falling asleep, while looking around the big, quiet
bedroom, Johannes lay thinking over these things. A night-light was
spluttering in a glass filled with equal parts of water and oil. As soon
as the flame was lighted, behind the milk-white, translucent shade
appeared strange, dreamy landscapes—formed by the unequal
thicknesses.
The chamber had an ancient, musty odor, and all the furniture bore
an old-fashioned stateliness. There was a queer pattern upon the
green bed-curtains, distressing to see; like half-opened eyes,
alternately squinting. The big bed-tassel hung down from above in
dogged dignity, like the tail of a lion keeping watch up above, on the
canopy of the four-poster.
Johannes felt very comfortable, yet there was something uncanny
around him that he did not quite relish. Once, it really seemed to be
the ponderous linen-chest of dark wood, with its big, brass-handled
drawers, upon which stood, under a bell-glass, a basket filled with
wax fruit. What the pictures represented could not be seen in the
dim light, but they were in the secret too, as was also the night-
stand with its crocheted cover, and the fearfully big four-poster.
Every half-hour "Cuckoo! Cuckoo!" rang through the house, as if
those out in the hall and in the vestibule were also in the secret; the
only one left out being the little fellow in clean underclothes and a
night-gown much too big for him, who lay there, wide awake,
looking around him. In the midst of all these solid, important, and
dignified things, he was a very odd and out-of-place phenomenon.
He felt that, in a polite way, he was being made sport of. Besides, it
remained to be seen whether, after his more or less unmannerly
adventures, he could ever be taken into confidence. Evidently the
entire house was, if not precisely hostile, yet in a very unfriendly
attitude. He kept his eye upon the bed-tassel, all ready to see the

57. lion wag his tail. In order to do that, however, he must surely first
become "converted," just like Aunt Seréna.
When the day dawned, this new life became more pleasant than he
had anticipated. Aunt Seréna presided at the breakfast, which
consisted of tea, fresh rolls, currant buns, sweet, dark rye-bread,
and pulverized aniseed. Upon the pier-tables, bright with sunshine,
stood jars of Japanese blue-ware, filled with great, round bouquets
of roses, mignonette, and variegated, ornamental grasses. The long
glass doors stood open, and the odor of new-mown grass streamed
in from the garden to the room, which was already deliciously
fragrant with the roses and mignonette, and the fine tea.
Aunt Seréna made no allusion to the foregoing day, nor to the death
of Johannes' father. She was full of kindly attentions, and
interrogated him affably, yet in a very resolute manner, concerning
what he had learned at school, and asked who had given him
religious instruction. It was now vacation time, and he might rest a
little longer, and enjoy himself; but then would come the school
again and the catechism.
Until now Johannes had had small satisfaction out of his solemn
resolution to value men more highly in order to live with them in a
well-disposed way. But this time he was more at ease. The nice, cool
house, the sunshine, the sweet smells, the flowers, the fresh rolls,
everything put him in good humor; and when Aunt Seréna herself
was so in harmony with her surroundings, he was soon prepared to
see her in the light of Daatje's glorification. He gazed confidingly into
the gleaming glasses of her spectacles, and he also helped her carry
the big, standing work-basket, out of which she drew the bright-
colored worsteds for her embroidery—a very extensive and
everlasting piece of work.

58. But the garden! It was a wonder—the joy of his new life. After being
released by his aunt until the hour for coffee, he raced into it like a
young, unleashed hound—hunting out all the little lanes, paths,
flower-plots, arbors, knolls, and the small pool; and then he felt
almost as if in Windekind's realm again. A shady avenue was there
which made two turns, thus seeming to be very long. There were
paths between thick lilac-bushes already in bloom; and there were
mock-oranges, still entirely covered with exceedingly fragrant white
flowers. There was a small, artificial hill in that garden, with a view
toward the west, over the adjacent nursery. Aunt Seréna was fond of
viewing a fine sunset, and often came to the seat on the hilltop.
There was a plot of roses, very fragrant, and as big as a plate. There
were vivid, fiery red poppies with woolly stems, deep blue larkspurs,
purple columbines, tall hollyhocks, like wrinkled paper, with their
strange, strong odor. There were long rows of saxifrage, a pair of
dark brown beeches; and everywhere, as exquisite surprises, fruit
trees—apples, pears, plums, medlars, dogberries, and hazel-nuts—
scattered among the trees which bore no fruit.
Indeed, the world did not now seem so bad, after all. A human
being—a creature admirably and gloriously perfect—a human
dwelling filled with attractive objects, and, close beside, a charming
imitation of Windekind's realm, in which to repose. And all in the line
of duty, with no departure from the prescribed path. Assuredly,
Johannes had looked only on the dark side of life. To confess this
was truly mortifying.
Towards twelve o'clock Daatje was heard in the cool kitchen, noisily
grinding coffee, and Johannes ventured just a step into her domain,
where, on all sides, the copper utensils were shining. In a little
courtyard, some bird-cages were hanging against the ivy-covered
walls. One large cage contained a skylark. He sat, with upraised
beak and fixed gaze, on a little heap of grass. Above him, at the top
of the cage, was stretched a white cloth.
"That's for his head," said Daatje, "if he should happen to forget he
was in a cage, and try to fly into the air."

59. Next to this, in tiny cages, were finches. They hopped back and
forth, back and forth, from one perch to another. That was all the
room they had; and there they cried, "Pink! Pink!" Now and then
one of them would sing a full strain. Thus it went the whole day
long.
"They are blind," said Daatje. "They sing finer so."
"Why?" asked Johannes.
"Well, boy, they can't see, then, whether it is morning or evening,
and so they keep on singing."
"Are you converted, too, Daatje?" asked Johannes.
"Yes, Master Johannes, that grace is mine. I know where I'm going
to. Not many can say that after me."
"Who besides you?"
"Well, I, and our mistress, and Dominie Kraalboom."
"Does a converted person keep on doing wrong?"
"Wrong? Now I've got you! No, indeed! I can do no more wrong. It's
more wrong even if you stand on your head to save your feet. But
don't run through the kitchen now with those muddy shoes. The
foot-scraper is in the yard. This is not a runway, if you please."
The luncheon was not less delicious: fresh, white bread, smoked
beef, cake and cheese, and very fragrant coffee, whose aroma filled
the entire house. Aunt Seréna talked about church-going, about the
choosing of a profession, and about pure and honest living.
Johannes, being in a kindly mood, and inclined to acquiescence,
avoided argument.

60. In the afternoon, as he sat dreaming in the shady avenue of lindens,
Aunt Seréna came bringing a tray, bearing a cooky and a glass of
cherry-brandy.
At half-past five came dinner. Daatje was an excellent cook, and
dishes which were continually recurring on stated days were
particularly well prepared. Vermicelli soup, with forced-meat balls,
minced veal and cabbage, middlings pudding with currant juice: that
was the first meal, later often recalled. Aunt Seréna asked a blessing
and returned thanks, and Johannes, with lowered eyes and head a
little forward, appeared, from the movement of his lips, to be doing
a little of the same thing.
Through the long twilight, Aunt Seréna and Johannes sat opposite
each other, each one in front of a reflector. Aunt Seréna was thrifty,
and, since the street lantern threw its light into the room, she was
not in a hurry to burn her own oil. Only the unpretending little light
for the making of the tea was glimmering behind the panes of milk-
white glass—with landscapes not unlike those upon the night-light.
In complete composure, with folded hands, sat Aunt Seréna in the
dusk, making occasional remarks, until Daatje came to inquire "if the
mistress did not wish to make ready for the evening." Then Daatje
wound up the patent lamp, causing it to give out a sound as if it
were being strangled. A quarter of an hour later it was regulated,
and, as soon as the cozy, round ring of light shone over the red
table-cover, Aunt Seréna said, in the most contented way: "Now we
have the dear little lamp again!"
At half-past ten there was a sandwich and a glass of milk for
Johannes. Daatje stood ready with the candle, and, upstairs, the
night-light, the chest of drawers with the wax fruit, the green bed-
curtains, and the impressive bed-tassel were waiting for him.
Johannes also descried something new—a big Bible—upon his night-
table. There was no appearance yet of any attempt at a
reconciliation on the part of the furniture. The cuckoo continued to
address himself exclusively to the stilly darkness, in absolute

61. disregard of Johannes; but the latter did not trouble himself so very
much about it, and soon fell fast asleep.
The morning differed but little from the foregoing one. Some Bibles
were lying ready upon the breakfast-table. Daatje came in, took her
place majestically, folded her half-bare wrinkled arms—and Aunt
Seréna read aloud. The day before, Aunt Seréna had made a
departure from this, her custom, uncertain how Johannes would take
it; but, having found the boy agreeable and polite, she intended now
to resume the readings. She read a chapter of Isaiah, full of harsh
denunciations which seemed to please Daatje immensely. The latter
wore a serious look, her lips pressed close together, occasionally
nodding her head in approval, while she sniffed resolutely. Johannes
found it very disconcerting, and could not, with his best endeavors,
keep his attention fixed. He was listening to the twittering of the
starlings on the roof, and the cooing of a wood-dove in the beech
tree.
In front of him he saw a steel engraving, representing a young
woman, clad in a long garment, clinging with outstretched arms to a
big stone cross that stuck up out of a restless waste of waters. Rays
of light were streaming down from above, and the young person
was looking trustfully up into them. The inscription below the
engraving read, "The Rock of Ages," and Johannes was deep in
speculation as to how the young lady had gotten there, and
especially how she was to get away from there. It was not to be
expected that she could long maintain herself in that uncomfortable
position—surely not for ages. That refuge looked like a peculiarly
precarious one; unless, indeed, something better might be done with
those rays of light.
Upon the same wall hung a motto, drawn in colored letters, amid a
superfluity of flowers and butterflies, saying: "The Lord is my
Shepherd. I shall not want."
This awakened irreverent thoughts in Johannes' mind. When the
Bible-reading was over, he was suddenly moved to make a remark.

62. "Aunt Seréna," said he, conscious of a rising color, and feeling rather
giddy on account of his boldness, "is it only because the Lord is your
Shepherd that you do not lack for anything?"
But he had made a bad break.
Aunt Seréna's face took on a severe expression, and adjusting her
spectacles somewhat nervously, she said: "I willingly admit, dear
Johannes, that in many respects I have been blessed beyond my
deserts; but ought not you to know—you who had such a good and
well-informed father—that it is very unbecoming in young people to
pass judgment, thoughtlessly, upon the lives of older ones, when
they know nothing either of their trials or of their blessings?"
Johannes sat there, deeply abashed, suddenly finding himself to be
a silly, saucy boy.
But Daatje stood up, and in a manner peculiarly her own—bending a
little, arms akimbo—said, with great emphasis: "I'll tell you what,
mistress! you're too good. He ought to have a spanking—on the
bare spanking place, too!" And forthwith she went to the kitchen.
VI
There were regularly recurring changes in Aunt Seréna's life. In the
first place, the going to church. That was the great event of the
week; and the weekly list of services and of the officiating clergymen
was devoutly discussed. Then the lace cap, with its silk strings, was
exchanged for a bonnet with a gauze veil; and Daatje was careful to
have the church books, mantle, and gloves ready, in good reason.
Nearly always Daatje went also; if not, then the sermon was
repeated to her in detail.
Johannes accompanied his aunt with docility, and tried, not without
a measure of success, to appreciate the discourse.

63. The visits of Minister Kraalboom were not less important. Johannes
saw, with amazement, that his aunt, at other times so stately and
estimable, now almost humbled herself in reverent and submissive
admiration. She treated this man, in whom Johannes could see no
more than a common, kindly gentleman, with a head of curling grey
hair, and with round, smoothly shaven cheeks, as if he belonged to a
higher order of beings; and the adored one accepted her homage
with candid readiness. The most delicious things the aunt had, in
fine wines, cakes, and liqueurs, were set before him; and, as the
minister was a great smoker, Daatje had a severe struggle with
herself after every visit, between her respect for the servant of the
Lord and her detestation of scattered ashes, stumps of cigars, and
tobacco-smelling curtains.
Once a week there was a "Krans," or sewing circle, and then came
Aunt Seréna's lady friends. They were more or less advanced in
years, but all of them very unprepossessing women, among whom
Aunt Seréna, with her erect figure and fine, pale face, made a very
good appearance; and she was clearly regarded as a leader. Puff-
cakes were offered, and warm wine or "milk-tea" was poured. The
aim of the gatherings was charitable. Talking busily, the friends
made a great many utterly useless, and, for the most part, tasteless,
articles: patchwork quilts, anti-macassars, pin-cushions, flower-pot
covers, picture frames of dried grasses, and all that sort of thing.
Then a lottery, or "tombola,"
[1]
as it was called, was planned for.
Every one had to dispose of tickets, and the proceeds were given,
sometimes to a poor widow, sometimes to a hospital, but more
often, however, to the cause of missions.
On such evenings Johannes sat, silent, in his corner, with one of the
illustrated periodicals of which his aunt had a large chestful. He
listened to the conversation, endeavoring to think it noble and
amiable; and he looked, also, at the trifling fingers. No one
interfered with him, and he drank his warm wine and ate his cake,
content to be left in peace; for he felt attracted toward none of the
flowers composing this human wreath.

64. But Aunt Seréna did not consider her duty accomplished in these
ways alone. She went out from them to busy herself in parish calls
on various households—rich as well as poor—wherever she thought
she could do any good. It was a great satisfaction to Johannes
when, at his request that he be allowed to go with her, she replied:
"Certainly, dear boy; why not?"
Johannes accompanied her this first time under great excitement.
Now he was going to be initiated into ways of doing and being good.
This was a fine chance.
So they set out together, Johannes carrying a large satchel
containing bags of rice, barley, sugar, and split peas. For the sick
there were jars of smoked beef and a flask of wine.
They first went to see Vrouw Stok, who lived not far away, in French
Lane. Vrouw Stok evidently counted upon such a visit, and she was
extremely voluble. According to her statements, one would say that
no nobler being dwelt upon earth than Aunt Seréna, and no nicer,
more grateful, and contented creature than Vrouw Stok. And
Dominie Kraalboom also was lavishly praised.
After that, they went to visit the sick, in reeking little rooms in
dreary back streets. And everywhere they met with reiterations of
gratitude and pleasure from the recipients, together with unanimous
praising of Aunt Seréna, until Johannes several times felt the tears
gather in his eyes. The barley and the split peas were left where
they would be of use, as were also the wine and the jars of smoked
beef.
Johannes and his aunt returned home very well pleased. Aunt
Seréna was rejoiced over her willing and appreciative votary, and
Johannes over this well-conducted experiment in philanthropy. If this
were to be the way, all would be well. In a high state of enthusiasm
he sped to the garden to dream away the quiet afternoon amid the
richly laden raspberry-bushes.

65. "Aunt Seréna," said Johannes, at table that noon, "that poor boy in
the back street, with the inflamed eyes and that ulcerated leg—is he
a religious boy?"
"Yes, Johannes, so far as I know."
"Then is the Lord his Shepherd, too?"
"Yes, Johannes," said his aunt, more seriously now, having in mind
his former remark. But Johannes spoke quite innocently, as if deep
in his own thoughts.
"Why is it, then, that he lacks so much? He has never seen the
dunes nor the ocean. He goes from his bed to his chair, and from his
chair to his bed, and knows only that dirty room."
"The Lord knows what is good for us, Johannes. If he is pious, and
remains so, sometime he will lack for nothing."
"You mean when he is dead?... But, Aunt Seréna, if I am pious I
shall go to heaven, too, shall I not?"
"Certainly, Johannes."
"But, Aunt Seréna, I have had a fine time in your home, with
raspberries and roses, and delicious things to eat, and he has had
nothing but pain and plain living. Yet the end is the same. That does
not seem fair, does it, Aunt Seréna?"
"The Lord knows what is good for us, Johannes. The most severely
tried are to Him the best beloved."
"Then, if it is not a blessing to have good things, we ought to long
for trials and privations?"
"We should be resigned to what is given us," said Aunt Seréna, not
quite at her ease.
"And yet be thankful only for all those delicious things? Although we
know that trials are better?"

66. Johannes spoke seriously, without a thought of irony, and Aunt
Seréna, glad to be able to close the conversation, replied:
"Yes, Johannes, always be thankful. Ask the dominie about it."
Dominie Kraalboom came in the evening, and, as Aunt Seréna
repeated to him Johannes' questions, his face took on the very same
scowl it always wore when he stood up in the pulpit; his wry mouth
rolled the r's, and, with the emphasis of delightful certainty, he
uttered the following:
"My dear boy, that which you, in your childlike simplicity, have asked,
is—ah, indeed—ah, the great problem over which the pious in all
ages have pondered and meditated—pondered and meditated. It
behooves us to enjoy gratefully, and without questioning, what the
good Lord, in His eternal mercy, is pleased to pour out upon us. We
should, as much as lies in our power, relieve the afflictions that He
allots to others, and at the same time teach the sufferers to be
resigned to the inevitable. For He knows what we all have need of,
and tempers the wind to the shorn lamb."
Then said Johannes: "So you, and Aunt Seréna, and I, have a good
time now, because we have no need of all that misery? And that sick
boy does need it? Is that it, Dominie?"
"Yes, my dear boy, that is it."
"And has Daatje, too, need of privations? Daatje said that she was
converted as completely as you and Aunt Seréna were."
"Daatje is a good, pious soul, entirely satisfied with what the Lord
has apportioned her."
"Yes, Dominie; but," said Johannes, his voice trembling with his
feeling, "I am not converted yet, not the least bit. I am not at all
good. Why, then, have I so much more given me than Daatje has?
Daatje has only a small pen, up in the garret, while I have the big
guest-room; she must do the scrubbing and eat in the kitchen, while

67. I eat in the house and get many more dainties. And it is not the Lord
who does that, but Aunt Seréna."
Dominie Kraalboom threw a sharp glance at Johannes, and drank in
silence, from his goblet of green glass, the fragrant Rhine wine. Aunt
Seréna looked, with a kind of suspense, at the dominie's mouth,
expecting the forthcoming oracle to dissipate all uncertainty.
When the dominie spoke again, his voice was far less kindly. He
said: "I believe, my young friend, that it was high time your aunt
took you home here. Apparently, you have been exposed to very bad
influences. Accustom yourself to the thought that older and wiser
people know, better than yourself, what is good for you; and be
thankful for the good things, without picking them to pieces. God
has placed each one in his station, where he must be active for his
own and his fellow-creatures' salvation."
With a sigh of contentment, Aunt Seréna took up her embroidery
again. Johannes was frightened at the word "picking," which brought
to mind an old enemy—Pluizer. Dominie Kraalboom hastened to light
a fresh cigar, and to begin about the "tombola."
That night, in the great bed, Johannes lay awake a long while,
uneasy and restless. His mind was clear and on the alert, and he
was in a state of expectancy. Things were not going right, though.
Something was the matter, but he knew not what. The furniture, in
the still night-time, wore a hostile, almost threatening air. The call of
the cuckoo spelled mischief.
About three or four o'clock, when the night-light had sputtered and
gone out, he lay still wider awake. He was looking at the bed-cord,
which, bigger and thicker than ordinary, was growing ominously
visible in the first dim light.

68. Suddenly—as true as you live—he saw it move! A slight quiver—a
spasmodic, serpentine undulation, like the tail of a nervous cat.
Then, very swiftly and without a rustle, he saw a small shadow drop
down the bed-cord. Was it a mouse?
After that he heard a thin little voice:
"Johannes! Johannes!"
He knew that voice. He lifted up his head and took a good look.
Seated upon the bed-tassel, astride the handle, was his old friend
Wistik.
He was the same old Wistik, looking as important as ever; yes, his
puckered little face wore a peculiar, almost frightened expression of
suspense. He was not wearing his little acorn-cup, but a smart cap
that appeared black in the twilight.
"I have news for you," cried Wistik. "A great piece of news. Come
with me, quick!"
"How do you do, Wistik?" whispered Johannes. He lay cozily
between the sheets, and was glad to see his friend again. Let the
chest of drawers and the cuckoo be as disagreeable as they wanted
to, now; here was his friend again. "Must I go with you? How can I?
Where to?"
"This way—up here with me," whispered Wistik. "I have found
something. It will make you open your eyes. Just give me your
hand. That's the best way. You can leave your body lying here while
you are away."
"That will be a fine sight," said Johannes.
But it happened without any trouble. He put out his hand, and in a
twinkling he was sitting beside Wistik, on the bed-tassel. And truly,
as he looked down below, there he saw his body lying peacefully fast
asleep. A ray of light streamed into the room, through the clover-leaf

69. opening in the blinds, and lighted up the sleeping head. Johannes
thought it an extremely pretty sight, and himself still a really nice
boy as he lay there among the pillows, with his dark curly hair about
the slightly contracted brows.
"Do you believe that I am very bad, Wistik?" said he, looking down
upon himself.
"No," said Wistik, "we must never fib to each other. Neither am I
bad; not a bit. I have found that out now, positively. Oh, I have
discovered so much since we last met! But we must not admire
ourselves on that account. That would be stupid. Come, now, for we
have not much time."
Together they climbed up the bed-cord. It was easy work, for
Johannes was light and small, and he climbed nimbly up the shaggy
rope. But it felt warm, and hairy, and alive in his hands!
Up they worked themselves, through the folds of the canopy. But the
bed-cord did not end there. Oh, no! It went on farther and grew
bigger and bigger, and then.... What they came to, I will tell you in
the following chapter.
[1] Lottery-Fair.
VII
It was, indeed, a real lion's tail, and not a bed-cord.
Johannes and Wistik were now sitting on the very back of the mighty
beast. Above them it was all dark, but out in front—away where the
lion was looking—the daylight could be seen.
They let themselves down cautiously to the ground. They were in a
large cave. Johannes saw streaks of water glistening along the rocky

70. walls.
Gently as they tried to slip past the monster, he yet discovered them,
and turned his shaggy head around, watching them distrustfully.
"He will not do anything," said Wistik. And the lion looked at them as
if they were a pair of flies, not worth eating up.
They passed on into the sharp sunlight outside, and, after several
blinding moments, Johannes saw before him a wide-spread, glorious
mountain view.
They were standing on the slope of a high, rocky mountain. Down
below, they saw deep, verdant valleys, whence the sound of
babbling brooks and waterfalls ascended.
In the distance was the dazzling, blinding glitter of sunshine upon a
sea of deepest, darkest blue. They could see the strand, and every
now and then it grew white with the combing surf. But there was no
sound; it was too far away.
Overhead, the sky was clear, but Johannes could not see the face of
the sun. It was very still all around, and the blue and white flowers
among the rocks were motionless. Only the rushing of the water in
the valleys could be heard.
"Now, Johannes, what do you say to this? It is more beautiful than
the dunes, is it not?" said Wistik, nodding his head in complete
satisfaction.
Johannes was enchanted at the sight of that vast expanse before
him, with the rocks, the flowers, the ravines, and the sea.
"Oh, Wistik, where are we?" asked he, softly, enraptured with the
view.
"My new cap came from here," said Wistik.
Johannes looked at him. The pretty cap that had appeared black in
the twilight proved to be bright red. It was a Phrygian cap.

71. "Phrygia?" asked Johannes, for he knew the name of those caps
well.
"Maybe," said Wistik. "Is not this a great find? And I know, too...."
Here he spoke in whispers again, very importantly, behind the back
of his hand, in Johannes' ear: "Here they know something more
about the little gold key, and the book, which we are both trying to
find."
"Is the book here?" asked Johannes.
"I do not know yet," said Wistik, a trifle disturbed. "I did not say
that, but the people know about it—that is certain."
"Are there people here?"
"Certainly there are. Human beings, and elves, and all kinds of
animals. And they know all about it."
"Is Windekind here, too, Wistik?"
"I do not doubt it, Johannes, but I have not seen him yet. Shall we
try to find him?"
"Oh, yes, Wistik! But how are we going to get down there? It is too
steep. We shall break our necks."
"No, indeed, if only you are not afraid. Just let yourself float. Then
you will be all right."
At first Johannes did not dare. He was wide awake, not dreaming;
and if any one wide awake were to throw himself down from a high
rock, he would meet his death. If one were dreaming, then nothing
would happen. If only he could know, now, whether he was awake
or dreaming!
"Come, Johannes, we have only a little time."
Then he risked it, and let himself drift downward. And it was
splendid—so comfortable! He floated gently down through the mild,

72. still air, arms and legs moving as in swimming.
"Is it only a dream, then?" he asked, looking down attentively at the
beautiful, blooming world below him.
"What do you mean?" asked Wistik. "You are Johannes, just the
same, and what you see, Johannes sees. Your body lies asleep, in
Vrede-best, at your aunt's. But did you ever in the daytime see
anything so distinct as this?"
"No," said Johannes.
"Well, then, you can just as well call your Aunt Seréna and Vrede-
best a dream—just as much as this."
A large bird—an eagle—swept around in stately circles, spying at
them with its sharp, fierce eyes.
Below, in the dark green of the valley, a small white temple, with its
columns, was visible. Close beside it a mountain stream tumbled
splashing down below. Still and straight as arrows, tall cypresses,
with their pale grey trunks and black-green foliage, encircled it. A
fine mist rose up from the splashing water, and, crowned with an
exquisite arc of color, remained suspended amidst the glossy green
myrtle and magnolia. Only where the water spattered did the leaves
stir; elsewhere everything was motionless.
But over all rang the warbling and chattering of birds, from out the
forest shade. Finches sang their fullest strains, and the thrushes
fluted their changeful tune, untiringly.
But listen! That was not a bird! That was a more knowing, more
cordial song; a melody that said something—something which
Johannes could feel, like the words of a friend. It was a reed, played
charmingly. No bird could sing like that.
"Oh, Wistik, who is playing? It is more lovely than blackbird or
nightingale."

73. "Pst!" said Wistik, opening his eyes wide. "That is only the flute, yet.
By and by you will hear the singing."
They sank down upon a mountain meadow, in a wide valley. The
limpid, blue-green rivulet flowed through the sunny grass-plot,
between blood-red anemones, yellow and white narcissi, and deep
purple hyacinths. On both sides of it were thick, round azalea-
bushes, entirely covered with fragrant, brick-red flowers. White
butterflies were fluttering back and forth across it. On the other side
rose tall laurel, myrtle, olive, and chestnut trees; and still higher the
cedars and pines—half-way up the mountain wall of red-grey
granite.
It was so still and peaceful and great blue dragon-flies with black
wings were rocking on the yellow narcissus flowers nodding along
the stream.
Then Johannes saw a fleeing deer, springing up from the sod in
swift, sinewy leaps; then another, and another.
The flute-playing sounded close by, but now there was singing also.
It came from a shady grove of chestnut trees, and echoed gloriously
from mountain-side to mountain-side, while the brook maintained
the rhythm with its purling, murmuring flow. The voices of men and
women could be heard, vigorously strong and sweetly clear; and,
intermingling with these somewhat rude shouts of joy, the high-
pitched voices of children.
On they came, the people, a joyous, bright-colored procession. They
all bore flowers—as wreaths upon their heads, as festoons in their
hands or about their shoulders-flute-players, men, women, and
children. And they themselves seemed living flowers, in their clear-
colored, charming apparel. They all had abundant, curling hair which
gleamed like dull gold in the sunshine, that tinted everything. Their
limbs and faces were tanned by the sun, but when the folds of their
garments fell aside, their bodies beneath them shone white as milk.
The older ones kept step, with careful dignity; the children bore little

74. baskets, with fruit, ribbons, and green branches; but the young men
and maidens danced as they went, keeping the rhythm of the music
in a way Johannes had never seen before. They swayed their bodies
in a swinging movement, with little leaps; sometimes even standing
still, in graceful postures, their arms alternately raised above their
heads, their loosened garments flowing free, and again arranging
themselves in charming folds.
And how beautiful they were! Not one, Johannes noted, old or
young, who had not those noble, refined features, and those clear,
ardent eyes, in which was to be found the deep meaning he was
always seeking in human faces—that which made a person instantly
his friend—that made him long to be cordial and intimate—that
which he had first perceived in Windekind's eyes, and that he missed
so keenly in all those human faces among which he had had to live.
That, they all had—man and woman, grey-haired one and little child.
"Oh, Wistik," he whispered, so moved he could scarcely speak, "are
they really human beings, and not elves? Can human beings be so
beautiful? They are more beautiful than flowers—and much more
beautiful than the animals. They are the most beautiful of all things
in this world!"
"What did I tell you?" said Wistik, rubbing his little legs in his
satisfaction. "Yes, human beings rank first in nature,—altogether
first. But until now we have had to do with the wrong ones—the
trash, Johannes—the refuse. The right ones are not so bad. I have
always told you that."
Johannes did not remember about it, but would not contradict his
friend. He only hoped that those dear and charming people would
come to him, recognize him as their comrade, and receive him as
one of them. That would make him very happy; he would love the
people truly, and be proud of his human nature.
But the splendid train drew near, and passed on, without his having
been observed by any one; and Johannes also heard them singing in

75. a strange, unintelligible language.
"May I not speak to them?" he asked, anxiously. "Would they
understand me?"
"Indeed, no!" said Wistik, indignantly. "What are you thinking about?
This is not a fairy tale nor a dream. This is real—altogether real."
"Then shall I have to go hack again to Aunt Seréna, and Daatje, and
the dominie?"
"Yes, to be sure!" said Wistik, in confusion.
"And the little key, and the book, and Windekind?"
"We can still be seeking them."
"That is always the way with you!" said Johannes, bitterly. "You
promise something wonderful, and the end is always a
disappointment."
"I cannot help that," said Wistik.
They went farther, both of them silent and somewhat discouraged.
Then they came to human habitations amid the verdure. They were
simple structures of dark wood and white stone, artistically
decorated and colored. Vines were growing against the pillars, and
from the roofs hung the branches of a strange, thickly leaved plant
having red flowers, so that the walls looked as if they were bleeding.
Birds were everywhere making their nests, and little golden statues
could be seen resting in marble niches. There were no doors nor
barriers—only here and there a heavy, many-colored rug hanging
before an entrance. It seemed very silent and lonely there, for
everybody was away; yet nothing was locked up, nor concealed. An
exquisite perfume was smoldering in bronze basins in front of the
houses, and columns of blue smoke coiled gently up into the still air.

76. Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.
More than just a book-buying platform, we strive to be a bridge
connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.
Join us on a journey of knowledge exploration, passion nurturing, and
personal growth every day!
ebookbell.com