Can I have checkpoints in Bayesian optimization for tuning hyperparameters of a neural network?

Question

I'm trying to implement Bayesian Optimization on a BiLSTM network. I'm planning to run this code in a university cluster but, they give us maximum of 2 days (48 hours) to run our job and if it goes beyond it, they automatically kill the job which probably will result in wasted time and resources for me and for other students waiting in que.   I was wondering if it would be possible to implement some kind of a checkpoint for bayesopt() to continue from where the job is left off: Basically, what I'm asking is if I can save my previous runs (variables bayesopt() observed) and load them in my next run and continue from where it stopped? I have not seen any documentation related to this (I may have missed it).   My understanding with bayesopt() is that, the more points are observed, the more accurate the answers bayesopt() gives. Is this right? If so, that means I might want to try to run it for more than 2 days maybe. The number of cores I can request are limited (the more I request, the longer I wait in que) and from what I'm estimating, the most complex combination of variables can take between 40 mins to 1 hour to train and give me a result ( obviously, not every combination will take this much time).

John Williams · Accepted Answer

Currently, there is no checkpointing argument. However, you can use the 'OutputFcn' argument along with the 'SaveFileName' argument to save to file, and the resume function to restart the process as follows,
 
 
x1 = optimizableVariable('x1',[-5,5]);
x2 = optimizableVariable('x2',[-5,5]);
fun = @rosenbrocks;

if exist('BayesoptResults.mat','file')
    load('BayesoptResults.mat');
    results = resume(BayesoptResults,...
    'SaveFileName', 'BayesoptResults.mat', ...
     'OutputFcn',{@saveToFile});
else
    results = bayesopt(fun, [x1, x2],'AcquisitionFunctionName',...
    'expected-improvement-plus', ...
    'SaveFileName', 'BayesoptResults.mat', ...
    'OutputFcn',{@saveToFile});
end


function f = rosenbrocks(x)
    f = 100*(x.x2 - x.x1^2)^2 + (1 - x.x1)^2;
end







Note that this saves to file on every iteration, so you might want to replace saveToFile with a custom function that saves occasionally, for performance reason.
The relevant docs are available here, resume and bayesopt.

Can I have checkpoints in Bayesian optimization for tuning hyperparameters of a neural network?

Need a Custom Version or Complete Simulation for This Problem?

How to Plot data in excel sheet using MATLAB ?

How can I use LSTM networks to solve a time series regression problem?

How to create an attention layer for deep learning networks?

Nvidia NVLink in MATALB using multiple Geforce 2080 ti cards

How do I build TensorFlow Lite for Deep Learning C++ code generation and deployment?

How to train labels for image blocks using neural network?

IEEE 13-BUS Hybrid AC & DC Power Distribution System using MATLAB Simulink

Adaptive Path Planning and Collision Avoidance for Autonomous Vehicles on Unstructured Indian Roads

Lithium-Ion Battery Electro-Thermal Modeling and Cooling Analysis in MATLAB

Get a Free Consultation or a Sample Assignment Review!