How to continue training after my jupyter notebook disconnected?
Can you provide more details?
Are you using Colab? If yes, Colab instances automatically terminate in case notebook is inactive for 90 minutes or after 12 hours.
Your training code runs from within Jupyter Notebook code block or from bash? In case your training requires longer hours, you should consider using other cloud alternatives like AWS spot instances.
Thank you for your response and Clouderizer app. Yes, I am using Colab just for starting the Clouderizer project. Then, I start Jupyter Notebook from the Clouderizer project and work on it to run FastAI’s solution for Kaggle competition, dog-breed-identification. Each of my training requires just 3-5 minutes to run within Clouderizer project. During this time I do nothing with Colab. But after around 1-1:30 hour of using Clouderizer, the Jupyter Notebook alerts, it’s just disconnected and I get Bad Gateway by refereshing the page or restarting the Jupyter Notebook. After trying 2-3 times yesterday I won’t be able to submit my solution to Kaggle competition.
It seems you are hitting Colab idle timeout limit.
Are you closing down Colab notebook browser tab? If yes, try to keep it open and see if it helps to overcome the idle timeout.
I do not close Colab notebook browser tab, but I do nothing with it after starting Clouderizer project with it. I’ll try it. Thank you so much.
I guess Google Colab, behind the scenes, uses Cloud Datalab framework from here.
According to this article, VM auto shutdown can be prevented by scrolling Colab notebook browser window. So i guess just switching to Colab notebook tab and scrolling it a bit every one hour should prevent Colab to idle timeout and VM to auto shutdown. Worth a try
Thank you so much. I did this in a different way by clicking on CONNECT button on the top right corner of the Colab every one hour;)
I Tried scrolling in google colab notebook but the it now runs for about an hour and then gives server connection error on the jupyter notebook.
For me the Jupyterlab gets disconnected if I start doing some heavy work like unzipping a big file or running large number of epochs . It generally lasts for 5-10 minutes . I’m facing this problem for the past few days . Earlier it was cool and running fine . Either I’m getting Error-404 or Error 502.
When you get this error, does Colab instance get terminated? You can verify this by running some simple command on Colab notebook like
This should give content of home folder. In case the instance is getting terminated, this folder will not have any Clouderizer project folder.
yes @prakash it does gets terminated on collab side and then I get this error in clouderizer JupyterLab side … How to resolve this ? Am I doing anything wrong.
I have also experienced this abrupt termination of Colab instances recently. It is generally visible during certain times of day, mostly when there is a high demand.
This can also be some other optimization from Google, reducing the idle timeout on their Colab notebook browser window.
In case I find something useful to handle this, I will post.
I would recommend exploring GCP. You get $300 worth credit on signup which can potentially give 1000+ hours of K80 GPU. You existing Clouderizer projects will run seamlessly on GCP machines. Here are the instructions
PS : We will soon be releasing our integration with GCP, using which you will be able to deploy your projects on GCP in a single click, all within Clouderizer console.
Google Collab is driving me nuts now. It keeps terminating every 10min.
Have to agree with this…this weekend my experience was the worst yet. I get bad gateway errors even when I’m actively working and my notebook changes aren’t being saved despite having my Google Drive account linked for weeks now. A !ls produces output of “sample_data” and a !pwd produces “/content”. My Clourderizer project says “Not Running” and I get “File Save Errors” when trying to save my notebook. I love the fact that Clouderizer and Google Colab are providing a free alternative for me to work with fast.ai with access to GPUs, but if these issues persist, it’s unusable for me.
GCP + Clouderizer is another great alternative to Google Colab. We worked hard last couple of months to make the integration as seamless as possible.
Give this a try and let me know your feedback.