Abstract
We propose a convex optimization approach to solving the nonparametric regression estimation problem when the underlying regression function is Lipschitz continuous. This approach is based on the minimization of the sum of empirical squared errors, subject to the constraints implied by Lipschitz continuity. The resulting optimization problem has a convex objective function and linear constraints, and as a result, is efficiently solvable. The estimated function computed by this technique, is proven to convergeto the underlying regression function uniformly and almost surely, when the sample size grows to infinity, thus providing a very strong form of consistency. Wealso propose a convex optimization approach to the maximum likelihood estimation of unknown parameters in statistical models, where the parameters depend continuously on some observable input variables. For a number of classical distributional forms, the objective function in the underlying optimization problem is convex and the constraints are linear. These problems are, therefore, also efficiently solvable.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Bazaara, M., Sherali, H., & Shetti, C. (1993). Nonlinear programming: Theory and algorithms. New York: Wiley.
Devroye, L.P. (1978). The uniform convergence of nearest neighbor regression function estimators and their application in optimization. IEEE Transactions on Information Theory, 24, 142–151.
Eubank, R. (1988). Spline smoothing and nonparametric regression. New York: M. Dekker.
Haussler, D. (1992). Decision theoretic generalization of the PAC model for neural net and other learning applications. Information and Computation, 100, 78–150.
Kolmogorov, A.N., & Tihomirov, V.M. (1961). ∈-entropy and ∈-capacity of sets in functional spaces. American Mathematical Translations, 17, 277–364.
Lee W., Bartlett, P., & Williamson, R. (1996). Efficient agnostic learning of neural networks with bounded fan-in. IEEE Transactions on Information Theory, 42, 2118–2132.
Pollard, D. (1984). Convergence of stochastic processes. Springer-Verlag.
Vapnik, V. (1996). The nature of statistical learning theory. New York: Springer-Verlag.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Bertsimas, D., Gamarnik, D. & Tsitsiklis, J.N. Estimation of Time-Varying Parameters in Statistical Models: An Optimization Approach. Machine Learning 35, 225–245 (1999). https://doi.org/10.1023/A:1007586831473
Issue Date:
DOI: https://doi.org/10.1023/A:1007586831473