Fred Roosta

Newton-MR: Inexact Newton Method with minimum residual sub-problem solver

Publisher: Elsevier BV

Date: 2022

DOI: 10.1016/J.EJCO.2022.100035

Inexact Nonconvex Newton-Type Methods

Publisher: Institute for Operations Research and the Management Sciences (INFORMS)

Date: 2021

DOI: 10.1287/IJOO.2019.0043

Abstract: The paper aims to extend the theory and application of nonconvex Newton-type methods, namely trust region and cubic regularization, to the settings in which, in addition to the solution of subproblems, the gradient and the Hessian of the objective function are approximated. Using certain conditions on such approximations, the paper establishes optimal worst-case iteration complexities as the exact counterparts. This paper is part of a broader research program on designing, analyzing, and implementing efficient second-order optimization methods for large-scale machine learning applications. The authors were based at UC Berkeley when the idea of the project was conceived. The first two authors were PhD students, the third author was a postdoc, all supervised by the fourth author.

Inexact Newton-CG algorithms with complexity guarantees

Publisher: Oxford University Press (OUP)

Date: 22-08-2022

DOI: 10.1093/IMANUM/DRAC043

Abstract: We consider variants of a recently developed Newton-CG algorithm for nonconvex problems (Royer, C. W. & Wright, S. J. (2018) Complexity analysis of second-order line-search algorithms for smooth nonconvex optimization. SIAM J. Optim., 28, 1448–1477) in which inexact estimates of the gradient and the Hessian information are used for various steps. Under certain conditions on the inexactness measures, we derive iteration complexity bounds for achieving $\\epsilon $-approximate second-order optimality that match best-known lower bounds. Our inexactness condition on the gradient is adaptive, allowing for crude accuracy in regions with large gradients. We describe two variants of our approach, one in which the step size along the computed search direction is chosen adaptively, and another in which the step size is pre-defined. To obtain second-order optimality, our algorithms will make use of a negative curvature direction on some steps. These directions can be obtained, with high probability, using the randomized Lanczos algorithm. In this sense, all of our results hold with high probability over the run of the algorithm. We evaluate the performance of our proposed algorithms empirically on several machine learning models. Our approach is a first attempt to introduce inexact Hessian and/or gradient information into the Newton-CG algorithm of Royer & Wright (2018, Complexity analysis of second-order line-search algorithms for smooth nonconvex optimization. SIAM J. Optim., 28, 1448–1477).

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Publisher: Cold Spring Harbor Laboratory

Date: 16-07-2022

DOI: 10.1101/2022.07.14.500142

Abstract: Trust and transparency are critical for deploying deep learning (DL) models into the clinic. DL application poses generalisation obstacles since training/development datasets often have different data distributions to clinical roduction datasets that can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models used to predict cancer of unknown primary with three independent RNA-seq datasets covering 10,968 s les across 57 primary cancer types. Our results highlight simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation (e.g., p-value = 0.0013 for calibration). Moreover, we demonstrate Bayesian DL substantially improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’ by designing a prototypical metric that evaluates the expected (accuracy) loss when deploying models from development to production, which we call the Area between Development and Production curve (ADP). In summary, Bayesian DL is a hopeful avenue of research for generalising uncertainty, which improves performance, transparency, and therefore safety of DL models for deployment in real-world.

University Of Queensland

Location: Australia

View Organisation

University Of British Columbia

Location: Canada

View Organisation

International Computer Science Institute

Location: United States of America

View Organisation

Efficient Second-order Optimisation Algorithms For Learning From Big Data

Start Date: 2018

End Date: 2020

Funder: Australian Research Council

Discovery Early Career Researcher Award - Grant ID: DE180100923

Start Date: 06-2018

End Date: 12-2023

Amount: $348,575.00

Funder: Australian Research Council

Linkage Projects - Grant ID: LP190100240

Start Date: 04-2021

End Date: 03-2025

Amount: $638,000.00

Funder: Australian Research Council

Industrial Transformation Training Centres - Grant ID: IC200100022

Start Date: 07-2021

End Date: 07-2026

Amount: $4,883,406.00

Funder: Australian Research Council