Escolar Documentos
Profissional Documentos
Cultura Documentos
3, 2009
UDC 519.237.5
A. S. Korkhin
The problem of estimating parameters of a linear regression with allowance for inequality constraints
on the parameters is considered in the special case when its variables have a trend. A parameter
estimation algorithm is described. The consistency of parameter estimates is proved and their
asymptotic distribution is found. Consistent estimates are proposed for the mean-square error matrix of
estimates of regression parameters and noise dispersion under rather general assumptions on the law
of noise distribution.
Keywords: linear regression, parameter, inequality constraint, estimation, mean-square error matrix,
noise dispersion.
In the present article, the results obtained in [1, 2] are applied to a regression with regressors having trends. The
regression being considered is of the form y t = xt a 0 + e t , t = 1, T , where y t R 1 , x t R n , a 0 R n , e t R 1 is noise, and
the symbol denotes transposition.
We estimate the parameter a 0 by solving the problem
S T (a ) =
1 T
( y t - xt a ) 2 min, g i ( a ) 0, i I ,
2 t =1
(1)
sup
dF t ( u ) 0 as
c .
T
We denote PT = x t xt and ET = diag ( rT11 , rT22 , K , rTnn ) , where rTii is an element on the main diagonal PT .
t =1
Assumption 2. The matrix PT is not degenerate for all T. The matrix RT = ET-1 PT ET-1 R as T , where R is
a positively definite matrix.
National Mining University, Dnepropetrovsk, Ukraine, korkhin@mail.ru. Translated from Kibernetika i Sistemnyi
Analiz, No. 3, pp. 5064, MayJune 2009. Original article submitted May 20, 2008.
1060-0396/09/4503-0373
373
Assumption 3. Constraints g i ( a ) are doubly continuously differentiable convex functions. The rank of a matrix G
whose rows are g i ( a 0 ) , i I , is completed.
Assumption 4. As T rTii , xT2 + 1,i / rTii 0, i = 1, n.
Assumption 5. A diagonal matrix ET ( m m ) with positive elements on the main diagonal eTi , i = 1, m , can be found
~
~
~
for which there exists the following limit of the matrix GT = ET GET-1 : lim GT = G, where G is a matrix composed of rows
T
~
~
~
g i ( a ), i = 1, n. At the same time, (1) the matrix G1 = lim GT 1 has a completed rank, where GT 1 = ET 1G1 ET-1 ,
0
ET 1 = diag ( eTi ),
i I 10 ,
-1
lim eTi eTj
< , i I , j = 1, n, where eTj = rTjj .
Assumption 5 holds true, in particular, when regressors are bounded. Then we have ET = T J m and ET = T J n .
It is assumed that independent variables are nonrandom.
COMPUTATION OF AN ESTIMATE OF THE REGRESSION PARAMETER VECTOR
Problem (1) is a convex programming problem that can be solved by a method from [3]. If the constraints in problem
(1) are linear, which is an important situation in actual practice, then problem (1) is a quadratic programming problem. To
solve it, methods of this class of problems are suitable. In particular, a method from [4, Ch. 23] can be used that is specially
adapted to the solution of problems of the form (1) with linear constraints.
Let us consider the case of linear constraints in more detail. In this case, the estimation problem after its
transformation assumes the form
(1/ 2) a PT a - a X T YT min, g i ( a ) = g i a - bi 0, i I ,
(3)
T T
~ r a
~ ~
~
(1/2) a
T - a d T min , Aa b.
(4)
T
T
T
T
~ R n -1 , ~
Here, a
PT = (x~t - x~T )( x~t - x~T ), dT = (x~t - x~T )( y t - yT ) , x~T = x~t / T , yT = y t / T , b R m , is
t =1
t =1
t =1
t =1
a vector whose components are bi , i I , and A is an m ( n - 1) matrix composed of (n -1) last columns of the matrix G,
whose rows are g i ( a 0 ) = g, i = 1, m.
Proof. We write a Lagrange function for minimization problem (3) in the form L( a, l ) = (1/ 2) a PT a - a X T YT
+ l (Ga - b ) , where l is an m-dimensional vector of Lagrange multipliers.
According to Assumption 2, necessary and sufficient conditions of the minimum of problem (3) are of the form
a L( a, l ) = PT a - X T YT + G l = On ,
(5)
l i ( g i a - bi ) = 0, l i 0, i = 1, m,
(6)
where a L( a, l ) it the gradient of the Lagrange function along the vector a, On is a zero n-dimensional vector, and l i
is the ith component of l.
374
Since no constraints are imposed on the free term a10 , the matrix G and its ith row g i are of the form
G = [Om M A ], g i = [ 0 A i ] ,
(7)
(8)
Let us consider condition (5). It is the representation of a system of n equations in vector form. We consider the first
of these equations that, with allowance for expression (8), is of the form
T
T
T
T
L( a, l )
= a1T + a 2 x t1 + a 2 x t 2 + + a n x t ,n -1 - y t = 0.
a1
t =1
t =1
t =1
t =1
(9)
Equation (9) must be satisfied by the sought-for estimates of parameters and, hence, it implies
~ x~ .
aT 1 = yT - aT 2 xT 1 - aT 3 xT 2 - - aTn xT ,n -1 = yT - a
T T
Thus, a formula for computation of estimates for free terms is proved.
Let us consider the ith equation (i = 2, n) in the system of equations (5),
T
T
T
T
T
L( a, l )
= a1 x ti + a 2 x ti x t1 + a 3 x ti x t 2 + + a n x ti x t , n -1 - x ti y t + a i l = 0, i = 2, n,
a i
t =1
t =1
t =1
t =1
t =1
j=2
t =1
t =1
t =1
t =1
(10)
j -1
t =1
t =1
t =1
- yT
t =1
t =1
t =1
j=2
t =1
t =1
i = 2, n.
We represent the obtained system of equations in the following vector form with allowance for the denotations of
problem (4):
~ - d + A l = O .
(11)
rT a
T
n
375
Let us consider condition (6). Taking into account the structure of the matrix G (see representation (7)), we obtain
~ - b ) = 0, l 0, i = 1, m.
l i ( Ai a
i
i
(12)
Since Eqs. (5) and (6) have a unique solution, Eqs. (11) and (12) obtained from them possess the same property. Then
these equations are the necessary and sufficient conditions of the minimum of problem (4) that are satisfied by the vector
~ of the vector of estimating the regression parameter a is the solution of problem (4).
~=a
~ . Hence, a subvector a
a
T
T
T
The theorem is proved.
As is well known [5, p. 24], the numerical solution of problem (4) is more exact than that obtained as a result of the
numerical solution of problem (3) (if no constraints are imposed on both problems). Under constraints, the accuracy of the
numerical solution of problem (4) will also be higher than that of problem (3) since the main error source lies in inversions
of matrices entering in the objective functions of the mentioned problems.
Based on Theorem 1, the estimation of the regression parameter can be reduced to the solution of a quadratic
programming problem in which moduli of elements of the matrix of its objective function are smaller than unities. To this
T
T
~
~, where B = s -1 s , s = diag ( s ), i = 1, n, s = T -1 ( y - y ) 2 , and s = T -1 ( x - x ) 2 ,
end, we put b = BT a
t
ti i
T
y x
x
xi
y
xi
t =1
t =1
i = 1, n. Denoting rTb = s x-1 rT s x-1 , dTb = ( s y s x ) -1 dT , and A b = ABT-1 , we obtain from problem (4) that
(1/2) b rTb b - b dTb min, A b b b.
An advantage of the solution of such an estimation problem (we denote this solution by b = bT ) is that bT does not
depend on the scale of measurement of variables. The components of bT are standardized estimates of (n -1) last components
of the vector a 0 . We call them beta weights by analogy with the term used in regression analysis without constraints. Such
weights can be conveniently used for estimation and comparison of the force of influence of independent variables on a
dependent variable.
PROPERTIES OF AN ESTIMATE OF A REGRESSION PARAMETER
We first prove the consistency of the solution of problem (1).
THEOREM 2. If Assumptions 15 are satisfied, then the solution aT of problem (1) is a consistent estimate of a 0 .
Proof. After transformation, problem (1) assumes the form
(1/ 2) a PT a - a X T YT min, g i ( a ) 0, i I,
(13)
where the denotations in the expression for the objective function are the same as in problem (3).
According to Assumption 2, the matrix PT is positively defined. Therefore, a nondegenerate matrix HT can always be
found such that we have PT = HT HT . We assume that b = HT a. Taking into account these transformations, quadratic
programming problem (13) is transformed into the form
(1/2) b b - b ( HT-1 ) X T YT min, hi ( b ) 0, i I ,
(14)
(15)
where bT* = ( HT-1 ) X T YT is the solution of problem (14) without taking into account the constraint b W b .
It follows from problem (15) that its solution bT is the projection of bT* onto W b . This projection is unique by virtue
of the convexity of W b . As is well known [6, p. 116], the distance from an arbitrary point A that does not belong to some
376
convex set to the projection of A onto this set does not exceed the distance from A to an arbitrary point belonging to the
mentioned set. Therefore, we have
(16)
|| bT - b 0 | |2 | | bT* - b 0 | |2 ,
where b 0 = HT a 0 W b since, with allowance for condition (2), we obtain
hi ( b 0 ) = hi ( HT a 0 ) = g i ( a 0 ) 0, i I .
From inequality (16) we have the truth of the inequality | | HT ( aT - a 0 )| |2 | | HT ( aT* - a 0 )| |2 , which implies
( aT - a 0 ) ET RT ET ( aT - a 0 ) ( aT* - a 0 ) ET RT ET ( aT* - a 0 ) . Then we obtain
m min ( RT )|| ET ( aT - a 0 )| |2 m max ( RT )| |UT* | |2 ,
where m max ( RT ) and
UT* = ET ( aT* - a 0 ) .
m min ( RT ) are,
respectively,
the
maximal
and
minimal
(17)
eigenvalues
of
RT
and
2
2
We denote eT2 ,min = min eTi
, where eTi
= rTii . Then the inequality | | ET ( aT - a 0 )| |2 eT2 ,min | | aT - a 0 | |2 holds true.
i =1, n
Taking into account this inequality, we obtain the following inequality from inequality (17):
m max ( RT )
|| aT - a 0 ||2
eT2 ,min m min ( RT
* 2
| |UT | | .
)
(18)
According to [7, p. 35], under Assumptions 1, 2, and 4, the distribution of UT* converges to the distribution of the
p
quantity U * ~ N (On , s 2 R -1 ). Therefore, based on Theorem XII [8, p.119], we have | |UT* | |2 | |U * | |2 , where the symbol
p
denotes the convergence in distribution. According to Assumption 2, the eigenvalues of the matrix RT converge to values
that are not equal to zero as T and, moreover, according to Assumption 4, eT2 ,min as T . Therefore, the first
multiplicand in the right side of inequality (18) converges in probability to zero. Then, by theorem X(a) [8, p. 118], the right
side of inequality (18) converges in probability to zero. This implies the truth of the statement of the theorem.
COROLLARY 1. If Assumptions 15 hold true, then the solution of problem (3) with linear constraints aT
converges in probability to a 0 .
COROLLARY 2. If Assumptions 15 are satisfied and, at the same time, we have ET = T J m and ET = T J n ,
then the solution of problem (3) with the linear constraints aT converges in quadratic mean to a 0 .
Proof. Under the condition of Corollary 2, aT* converges in quadratic mean to a 0 [9, p. 43], m max ( RT ) m max ( R ),
and m min ( RT ) m min ( R ) > 0 as T . This directly implies the statement formulated.
Let us determine the limiting distribution law for the estimate of the parameter of the linear regression. The proof
scheme will be similar to that described in [1]. However, it differs from the proof presented in this article in connection with
the presence of a trend in regressors. Let us consider three auxiliary results.
LEMMA 1. Let the conditions of Theorem 2 be satisfied. Then, for a given number d > 0, numbers e > 0 and T0 > 0
can be found such that we have
P{||UT || e} < d, T > T0 , UT = ET ( aT - a 0 ).
(19)
(20)
Theorem 2, it is shown that ||UT* ||2 ||U * ||2 . Then we have mT | |UT* | |2 m | |U * | |2 . Therefore, taking into account
377
inequality (20), for a given number d > 0, some numbers e > 0 and T0 > 0 can be found for which the following chain of
inequalities takes place: d > P{mT ||UT* ||2 e} > P{| |UT | |2 e}, T > T0 . The lemma is proved.
LEMMA 2. The quadratic programming problem
(1/2) a Ra - Q a min, Ga b,
(21)
has a solution a(G , Q ) that is continuous with respect to G and Q. Here, a, Q R n, R is a positively definite matrix of
order n, G is an m n matrix, and b R m .
Proof. As is shown in proving Theorem 2, by virtue of the positive definiteness of the matrix R, the representation
R = H H is correct, where H is a nondegenerate matrix. We put b = Ha, P = ( H -1 ) Q , and S = GH -1 . Taking into account
these transformations, quadratic programming problem (21) is transformed into the form
min{(1/ 2) b b - P b | Sb b}.
(22)
We denote its solution by b( S, P ). For arbitrary Q1 and Q2 = Q1 + DQ and G1 and G2 = G1 + DG, we have
P2 = P1 + DP and S 2 = S 1 + DS , where Pi = H -1Qi and S i = H -1Gi , i = 1, 2. From this we obtain
DQ = HDP , DG = HDS .
(23)
(24)
where Da = a(G2 , Q2 ) - a(G1 , Q1 ) and Db = b( S 2 , P2 ) - b( S 1 , P1 ). Let DQ On , and let DG Omn , which, by virtue
of the nondegeneracy of H and expressions (23), implies that DS Omn and DP On . Then, according to a lemma
from [10], we have Db On . This and expression (24) imply that Da On .
The lemma is proved.
To obtain further results, the left side of the ith constraint g i ( a ) should be expanded in the vicinity of a = a 0 by its
Taylor series expansion as follows:
1
g i ( a ) = g i ( a 0 ) + g i ( a 0 ) + ( a - a 0 ) g i 2 ( a 0 + q 1 Da ).
2
(25)
0
1
0
b i ( X ) = ( eTi g i ( a )ET ) X + li ( X ) Om 1 , i I 1 ,
0
0
0
1
b i ( X ) = eTi g i ( a ) + ( eTi g i ( a )ET ) X + li ( X ) Om 2 , i I 2 ,
jT ( X ) =
1
X RT X - QT X min,
2
(26)
b i ( X ) = eTi g i ( ET-1 X + a 0 ), i I ,
(27)
1
(28)
X ( eTi ET-1 g i 2 ( a 0 + q 1 ET-1 X ) ET-1 ) X , i I ,
2
where the function g i 2 and the value of q 1 are mentioned in series (25).
Proof. Problem (26) has a strictly convex objective function (according to Assumption 2) and a convex accessible
region since its constraints are convex functions. Hence, the solution of problem (26) (we denote it by UT* ) is unique. It
li ( X ) =
378
*
*
*
RT UT* - QT + b i (UT* ) uTi
= On , uTi
b i (UT* ) = 0, uTi
0, i I ,
(29)
i =1
*
where uTi
is the Lagrange multiplier corresponding to the ith constraint.
Problem (1) also is a convex programming problem and the necessary and sufficient conditions of its minimum are
satisfied by aT ,
m
(30)
PT aT - aT X T YT + g i ( aT ) l Ti = On , l Ti g i ( aT ) = 0, l Ti 0, i I .
i =1
(31)
i =1
*
Comparing expressions (29) and (31), we obtain that condition (29) holds true when UT* = UT and uTi
= uTi 0,
i I , since, according to expression (27), we have b i (UT ) = eTi g i ( aT ) and b i (UT ) = eTi ET-1g i ( aT ), i I . Since
conditions (29) determine a unique solution of problem (26), the lemma is true.
Let us define the limit for UT .
THEOREM 3. If Assumptions 15 hold true, then, as T , the random quantity UT = ET ( aT - a 0 ) converges in
distribution to a random quantity U, i.e., to the solution of the problem
~
jT ( X ) = (1/ 2) X RX - Q X min, G1 X Om 1 ,
(32)
~
~
where the ( m1 n ) matrix G1 consists of rows of the matrix G whose indices are i I 10 .
Proof. Let us consider the quadratic programming problem
~
jT0 ( X ) = (1/ 2) X RX - QT X min, GT 1 X Om 1 .
(33)
We denote its solution by UT0 . By Theorem 2.6.1 [7, p. 35], for the vector QT , we have
p
QT Q ~ N (On , s 2 R ), T .
(34)
~
~
According to Lemma 2, UT0 is a continuous function of QT and GT 1 , UT0 = f (QT , GT 1 ). From this and limit (34) and
p
~
~
~
~
also taking into account that, according to Assumption 5, we have lim GT 1 = G1 , we obtain f (QT , GT 1 ) f (Q , G1 ) .
T
~
According to problem (32), we have U = f (Q , G1 ) . Thus, as T , we have
p
UT0 U .
(35)
According to Assumption 2, jT0 ( X ) is a strongly convex function with respect to X . It possesses the property [11, p. 54]
||UT - UT0 ||2
2 0
[ jT (UT ) - jT0 (UT0 )], m > 0,
m
(36)
since UT O, i.e., UT satisfies the constraints of problem (33) (the accessible region of this problem includes the
accessible region of problem (26)). According to property (36), for an arbitrarye > 0, we have
2
P{ | jT0 (UT ) - jT (UT )| < ( e1 / 2)} + P{[ jT (UT ) - jT0 (UT0 )] < ( e1 / 2)}- 1 ,
(37)
where e1 = e 2 m / 2 .
To further transform the right side of inequality (37), we derive some relationships connecting objective functions in
problems (26) and (33). For an arbitrary e 2 > 0, we have
P{| jT0 ( X ) - jT ( X )| < e 2 } = P{|1/ 2X ( R - RT ) X | < e 2 }.
(38)
(39)
where a is some positive number. According to Assumption 2, for a sufficiently large T, the third addend in the right
side of inequality (39) equals 0. Then, applying Lemma 1, from inequality (39) we obtain
P{ | jT0 (UT ) - jT (UT )| < e 2 } 1- d, T > T1 .
(40)
In relationship (38), we put X = UT0 . After performing similar calculations and taking into account that UT0 has a limit
distribution (according to convergence (35)), for arbitrary numbers e 2 > 0 and d > 0, we obtain
P{ | jT0 (UT0 ) - jT (UT0 ) | < e 2 } = DT 2 1- d, T > T2 .
(41)
We consider the second addend in the right side of relationship (37). After some simple transformations, we have
P{[ jT (UT ) - jT0 (UT0 )] < ( e1 / 2)} P{jT (UT ) - jT (UT0 ) 0}
+ P{ | jT (UT0 ) - jT0 (UT0 ) | < ( e1 / 2)} - 1 = DT 1 + DT 2 - 1,
(42)
(43)
0
Here, we have ETk = diag ( eTi ), i I k .
According to expressions (28) and (35) and Assumption 5, we have p lim li (UT0 ) =
T
380
1 0
(U ) ( eTi ET-1 g i 2 ( a 0
2 T
According to expression (42), we obtain DT 1 P{UT0 OT }. Therefore, we have the chain of inequalities
~
DT 1 = P{jT (UT ) - jT (UT0 ) 0} P{GT 1UT0 + L1 (UT0 ) Om 1 ,
~
~
GT 2UT0 + L2 (UT0 ) + ET 2 g ( 2 ) ( a 0 ) Om 2 } P{GT 1UT0 + L1 (UT0 ) Om 1 }
~
+ P{GT 2UT0 + L2 (UT0 ) + ET 2 g ( 2 ) ( a 0 ) Om 2 } - 1
0P{yTi(1) + li (UT0 ) 0}
iI 1
(2 )
+ P{ yTi
+ li (UT0 ) + eTi g i ( a 0 ) 0} - ( m1 - 1) - ( m2 - 1) -1
iI 20
iI 1
iI 2
~
(k )
is the ith component of the vector GTkUT0 , k = 1, 2 .
where yTi
p ~
~
~
According to problem (32), we have G1U Om 1 . Then expressions (35) and (44) imply that GT 1UT0 G1U Om 1 .
Since p lim li (UT0 ) = 0, i I , it is easy to show that, for an arbitrary d1 > 0, there is some T3 > 0 for which we have
T
(1)
P{yTi
+ li (UT0 ) 0} > 1- d1 , i I 10 , T > T3 .
By virtue of the convergence of li (UT0 ), i I , to zero in probability, from expressions (34) and (43) and
Assumption 5, for an arbitrary d 2 > 0, we have
P{e -1 (| yT( 2 ) | + li (UT0 )) - g i ( a 0 )} > 1- d 2 , i I 20 , T > T4 .
Ti
Substituting this inequality and the previous inequality in the lower estimate obtained for the quantity D1T , we obtain
DT 1 1- m1 d1 - m2 d 2 , T > max(T3 , T4 ) .
(45)
We put e 2 = e1 / 2 in expression (41). Then, with allowance for inequality (45), from relationship (42) we obtain
P{[ jT (UT ) - jT0 (UT0 )] <
e1
} > 1- d - m1 d1 - m2 d 2 , T > max(T2 , T3 , T4 ).
2
(46)
Substituting inequality (40) when e 2 = e1 / 2 and inequality (46) in relationship (37), we have P{| |UT -UT0 | |2
< e 2 } > 1- 2d - m1 d1 - m2 d 2 , T > max Ti . Thus, we obtain
1 i 4
The statement of the theorem follows from this expression and expression (35).
DETERMINATION OF THE ACCURACY OF ESTIMATION
OF REGRESSION PARAMETERS
In this section, we determine the accuracy of estimation for problem (3). We consider that the noise in the model is
normally distributed, i.e., it satisfies the assumption that is given below and is a special case of Assumption 1.
Assumption 6. The random quantities e t in the regression equation are independent and normally distributed with
zero expectations and dispersions s 2 .
381
Let us consider main regression accuracy characteristics such as estimates of the noise dispersion and mean-square
error (m.s.e.) matrix of parameter estimates.
Auxiliary results. For understanding further calculations, we consider the concepts of active and inactive constraints
to within some positive number x such that we have - x > g i ( a 0 ), i I 20 [2]. The first form of a constraint (let its number be i)
satisfies the condition - x g i ( aT ) 0. Accordingly, the ith inactive constraint satisfies the inequality g i ( aT ) < - x. In [2],
a random quantity g Ti is introduced such that, for a finite sample with a number of observations T, we have
1 if the ith constraint is active to within x ,
g Ti =
0 if the ith constraint is inactive to within x.
As is obvious, quantities g Ti , i I , are functions of aT .
The results presented below will be used in what follows.
LEMMA 4 [2, Theorem 3]. Let aT be a consistent estimate. In this case, if the ith constraint is active when a = a 0
(i I 10 ), then we have p lim g Ti = 1and if the ith constraint is inactive when a = a 0 (i I 20 ), then we have p lim g Ti = 0.
T
T
n
LEMMA 5 [12, Lemma 2]. Let S ( R , B , b, s , N ) be a vector function of variables s R , s > 0; N R , where R is
1
si (R , B , b, s , z ) s j (R , B , b, s , z ) f ( z , R ) dz
Rn
s T2 = (T - n + g Ti ) -1 ( y t - xt aT ) 2 .
iI
(47)
t =1
The consistency of estimate (47) for various noise distributions e t , t = 1,2, K , and regressors with trends is determined
by the following theorem.
THEOREM 4. If Assumptions 15 are satisfied, then s T2 is a consistent estimate of s 2 .
Proof. Transforming the expression for s T2 , we have
T
s T2 =
( yt - xt a 0 )2
UT RT UT - 2UT QT t =1
+
T - n + g Ti
iI
T
,
T - n + g Ti
(48)
iI
where the random quantity QT is defined in problem (26) and UT = ET ( aT - a 0 ) . The matrix ET is determined by
Assumption 2.
Under Assumptions 1 and 2, limit (34) takes place. By Theorem 3 and by virtue of Assumptions 15, UT has a limit
distribution. Hence, the numerator of the first addend in expression (48) has a limit distribution. Its denominator tends to
infinity since, according to Lemma 4, the sum gT i converges in probability to a finite number m1 . Then, based on
iI
Theorem X(a) [8, p. 118], the first addend in expression (48) converges in probability to zero. According to Assumption 1
and Lemma 4, the second addend converges to s 2 by the law of large numbers.
The theorem is proved.
382
With a view to obtaining an expression for the m.s.e. matrix of regression parameter estimates K T0 for a finite T, we
write problem (3) after transformation in the form
~
(1/ 2)Y RT Y - Y QT min, GT Y ET ( b - Ga 0 ) ,
where X = ET ( a - a 0 ) and b is a vector whose components are bi , i I .
Putting x = X / s, we write this problem in the form
~
~
(1/ 2) x RT x - x qT min, GT 1 x BT(1) , GT 2 x BT( 2 ) .
(49)
Here, we have qT = QT / s ~ N (On , RT ) (according to the expression QT = ET-1 e t x t and Assumption 6). It follows
t =1
qT Q / s = q ~ N (On , R ), T .
(50)
The right sides of constraints in problem (49) are defined by the expressions
BT( i ) = ETi ( b ( i ) - Gi a 0 ) / s, i = 1, 2 ,
(51)
where b ( i ) R m i has components b j , j I i0 (see problem (3)) and, at the same time, we have b (1) = G1 a 0 . Thus, we
obtain BT(1) = Om 1 , and components of BT( 2 ) are nonnegative.
We specify the m.s.e. matrix of regression parameter estimates K T0 by the expression
K T0 = M {UT UT } = s 2 kT0 , kT0 = M {uT uT },
(52)
~ ~
~
~
where uT = UT / s = S ( RT , GT 1 , GT 2 , BT(1), BT( 2 ), qT ) is the solution of problem (49) and kT0 = M {S ( RT , GT 1 , GT 2 , bT(1) ,
~
~
bT( 2 ), qT )S ( RT , GT 1 , GT 2 , bT(1), bT( 2 ), qT )} .
It follows from Theorem 3 that uT converges in distribution to the random quantity u = s ( R , b (1) , q ) that is the
solution of the problem
~
(1/ 2) x Rx - q x min, G1 x b (1) = Om 1 ,
(53)
(54)
nS i ( RT , GT 1 , GT 2 , bT
(1)
, bT( 2 ), z )
~ ~
S j ( RT , GT 1 , GT 2 , bT(1), bT( 2 ) , z ) f ( z , RT )dz.
(55)
383
~ ~
~ ~
Here, S i ( RT , GT 1 , GT 2 , bT(1) , bT( 2 ), z ) is the ith component of S ( RT , GT 1 , GT 2 , bT(1) , bT( 2 ), z ) , i.e., of the solution of
the problem
~
~
(1/ 2) x RT x - x qT min, G1T x bT(1), G2T x bT( 2 ) .
(56)
~
The matrices GTk , k = 1, 2 , appearing in expression (55) are defined in inequality (43), vectors bT( k ) R m k have
components bTi , and, at the same time, we have bT( k ) = [ bTi ] , i I k0 , k = 1, 2 ,
bTi =
eTi ( bi - ai aT )
(1- g Ti ), i I ,
sT
(57)
where eTi is an element on the main diagonal of the matrix ET specified by Assumption 5.
By virtue of the consistency of aT (Theorem 2) and s T2 (Theorem 4) and with allowance for Lemma 4, we have
p lim bT(1) = b (1) = Om 1 , bT( 2 ) as T .
T
(58)
Problem (56) differs from problem (49) in the right sides of its constraints. According to expression (57), they should
be modified to obtain a consistent estimate of the m.s.e. matrix of regression parameter estimates. As such a matrix, we
consider the matrix
(59)
K T = s T2 kT ,
~ ~
where the quantity s T2 is defined by expression (47), kT = [ K ijT ( RT , GT 1 , GT 2 , bT(1) , bT( 2 ) )], i, j = 1, n, and, at the same
~ ~
time, K Tij ( RT , GT 1 , GT 2 , bT(1), bT( 2 ) ) is specified by expression (55).
Let us show that the matrix K T specified by expression (59) is a consistent estimate of the m.s.e. matrix of regression
parameter estimates.
THEOREM 5. If Assumptions 26 are satisfied, then we have p lim K T = K , where the matrix K is defined by
T
(60)
where k = K / s 2 = M {UU } / s 2 .
To this end, we consider the quadratic programming problem
~
(1/ 2) x RT x - qT x min, GT 1 x bT(1) ,
(61)
~
where GT 1 is a matrix (of a completed rank) of dimension m1 n (according to Assumption 3).
~
We denote the solution of problem (61) by s ( RT , GT 1 , bT(1) , qT ). We introduce the matrix (of order n) of second
moments of this solution by defining its ( i, j )th element as follows:
~
k Tij ( RT , GT 1 , bT(1) ) =
nsi ( RT , GT 1 , bT
~
, z ) s j ( RT , GT 1 , bT(1), z ) f ( z , RT ) dz.
(1)
(62)
~
~
Here, si ( RT , GT 1 , bT(1), z ) is the ith component of s ( RT , GT 1 , bT(1), qT ) when qT = z and f ( z , RT ) is the density of
distribution of qT ,
(63)
~
~
Since we have k = M {s( R , G1 , b (1) , q ) s ( R , G1 , b (1), q )} and, according to limit (50), q ~ N (On , R ), an element ( i, j ) of
the matrix k assumes the form
384
~
k ij ( R , G1 , b (1) ) =
nsi ( R , G1 , b
~
, z ) s j ( R , G1 , b (1) , z ) f ( z , R ) dz, i, j = 1, n,
(1)
(64)
~
where si ( R , G1 , b (1), z ) is the ith component of u when q = z and f ( z , R ) is the density of distribution of q,
f ( z , R ) = ( 2p ) - n / 2 (det R ) -1/ 2 exp ( - (1/ 2) z R -1 z ).
(65)
~
~
According to Assumption 2 and limits (44) and (58), we have lim RT = R, lim GT 1 = G1 , and p lim bT(1) = b (1) = Om 1 .
T
Then, applying Lemma 5 to expression (62), from expressions (62) and (64), we obtain
~
~
p lim k Tij ( RT , GT 1 , bT(1) ) = k ij ( R , G1 , b (1) ) , i, j = 1, n.
(66)
~
~ ~
We now show that k Tij ( RT , GT 1 , bT(1) ) and K Tij ( RT , GT 1 , GT 2 , bT(1), bT( 2 ) ) (see expression (55)) have a common limit
as T . We introduce the following set:
~
~
w(GT 1 , bT(1), bT( 2 ) ) = {z: s( RT , GT 1 , bT(1), z ) bT( 2 ), z R n }.
~
~
It is easy to make sure that we have w(GT 1 , bT(1), bT( 21) ) w(GT 1 , bT(1), bT( 22) ) if the inequality bT( 21) bT( 22) is true.
According to limit (58), for arbitrary numbers e > 0, d > 0, M > 0, and h > 0, some T0 > 0 can be found such that we
have
P{ || bT(1) || e} > 1- d, P{| | bT( 2 ) | | M } > 1- h, T > T0 .
Putting
(67)
~
~
~
j ij ( RT , GT 1 , bT(1), z ) = si ( RT , GT 1 , bT(1), z ) sj ( RT , GT 1 , bT(1), z ) ,
~
~
~
F ij ( RT , GT , bT(1), bT( 2 ), z ) = S i ( RT , GT , bT(1) , bT( 2 ), z ) S j ( RT , GT , bT(1), bT( 2 ), z ) ,
~
~ ~
and
S k ( RT , GT , bT(1), bT( 2 ), z ) = S k ( RT , GT 1 , GT 2 , bT(1), bT( 2), z ), k = i, j,
~
~
= K Tij ( RT , GT 1 , GT 2 , bT(1), bT( 2 ) ), we obtain from expressions (55) and (62) that
where
~
~
| K Tij ( RT , GT , bT(1) , bT( 2 ) ) - k Tij ( RT , GT 1 , bT(1) )| =
j ij
( 1) ( 2)
~
denoting K Tij ( RT , GT , bT(1), bT( 2 ) )
( RT , GT 1 , bT(1), z ) f ( z , RT ) dz
w ( bT , bT )
(1)
j ij ( RT , GT 1 , bT(1), z ) f ( z , RT ) dz -
R n \ w ( bT , bT( 2) )
( 1)
~
F ij ( RT , GT , bT(1), bT( 2 ), z ) f ( z , RT ) dz
w( bT , bT( 2) )
(1)
~
F ij ( RT , GT , bT(1), bT( 2 ), z ) f ( z , RT ) dz
R n \ w ( bT , bT( 2) )
=
R
(1)
| j ij ( RT , G~T 1 , bT(1),
~
~
x ) - F ij ( RT , GT , bT(1), bT( 2 ), z )| f ( z , RT ) dz = LT ( RT , GT , bT(1), bT( 2 ) )
(68)
\ w ( bT , bT( 2) )
~
~
~
since we have si ( RT , GT 1 , bT(1) , z ) = S i ( RT , GT , bT(1) , bT( 2 ) , z ), i = 1, n, for z w(GT 1 , bT(1) , bT( 2 ) ) .
~
~
For a fixed z, we have S i ( RT , GT , bT(1), bT( 2 ), z ) si ( RT , GT 1 , bT(1), z ) as T since bT( 2 ) . Therefore, for an
arbitrary g > 0, some numbers M > 0 and N > 0 can be found such that the following inequality is true:
385
max
||bT( 1) || e ,
||bT( 2) || M
(1)
| j ij ( RT , G~T 1 , bT(1),
~
z ) - F ij ( RT , GT , bT(1), bT( 2 ), z )| f ( z , RT ) dz g,
R n \ w ( bT , bT( 2) )
~ ~
||GT -G || N
where e > 0 is given. Then, taking into account inequality (67), we obtain
~
P{LT ( RT , GT , bT(1), bT( 2 ) ) g} P{| | bT(1) | | e, | | bT( 2 ) | | M ,
~
~
||GT - G | | N }
~
~
~
~
= P{|| bT(1) || e, ||GT - G || N } - P{| | bT(1) | | e, | |GT - G | | N , | | bT( 2 ) | | < M }
~
~
P{| | bT(1) || e, ||GT - G || N } - P{|| bT( 2 ) | | < M } P{| | bT(1) | | e} -P{| | bT( 2 ) | | < M } > 1- d - h, T > T0 .
From expression (68) and the latter inequality, we have
~
~
P{| K Tij ( RT , GT , bT(1), bT( 2 ) ) - k Tij ( RT , GT 1 , bT(1) )| g} 1- d - h, T > T0 .
Thus, we obtain
~
~
p lim | K Tij ( RT , GT , bT(1), bT( 2 ) ) - k Tij ( RT , GT 1 , bT(1) )| = 0.
T
(69)
~
~
Expressions (69) and (66) imply that K Tij ( RT , GT , bT(1), bT( 2 ) ) converges in probability to k ij ( R , G1 , b (1) ), which
proves the validity of relationship (60). Then, with allowance for Theorem 4, we obtain the statement of this theorem.
REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
386
A. S. Korkhin, Some properties of regression parameters estimates under a priori inequality constraints,
Kibernetika, No. 6, 106114 (1985).
A. S. Korkhin, Determining sample characteristics and their asymptotic linear-regression properties estimated using
inequality constraints, Cybernetics and Systems Analysis, No. 3, 445456 (2005).
A. S. Korkhin, Solution of problems of a nonlinear least-squares method with nonlinear constraints based on the
linearization method, Probl. Upravl. Inform., No. 6, 124135 (1998).
C. Lawson and R. Henson, Numerical Solution of Problems by the Method of Least Squares [Russian translation],
Nauka, Moscow (1986).
J. Maindonald, Computational Algorithms in Applied Statistics [Russian translation], Finansy i Statistika, Moscow
(1988).
B. T. Polyak, Introduction to Optimization [in Rusian], Nauka, Moscow (1983).
T. W. Anderson, Statistical Analysis of Time Series [Russian translation], Mir, Moscow (1976).
S. R. Rao, Linear Statistical Models and Their Applications [Russian translation], Nauka, Moscow (1968).
E. Z. Demidenko, Linear and Nonlinear Regressions [in Russian], Finansy i Statistika, Moscow (1981).
M. Z. Khenkin and E. I. Volynskii, A search algorithm to solve the general mathematical programming problem,
Zh. Vychisl. Mat. Mat. Fiz., No. 1, 6171 (1976).
V. G. Karmanov, Mathematical Programming [in Russian], Nauka, Moscow (1975).
A. S. Korkhin, Parameter estimation accuracy for nonlinear regression with nonlinear constraints, Cybernetics and
Systems Analysis, No. 5, 663672 (1998).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.