How to auto-restart the server after crash
Due to various reasons it may or may not happen that your server randomly crashes. Either because of a game bug, a mod incompatibility, a plugin issue, you name it.
Most of the time, you will not be able to restart the server manually right away. Since everyone has to sleep, work or is on vacation at times you could end up with a very unhappy community. So here are a few tips on how to prevent that from happening.
The right settings in your Torch.cfg
Torch itself can handle quite a lot of crashes by itself just fine. However it does not necessarily do it on its own.
Make sure you have your TickTimeout and RestartOnCrash settings set up. It is recommended to have them on:
RestartOnCrash allows torch to just kill the server instance and restart it when it detects the game to be broken.
TickTimeout monitors the games Threads and when it notices one to be stuck for more than the configured amount of seconds it restarts the server after killing it first.
Also its recommended to set:
This causes the server to instantly start up after opening the Torch.Server.exe file. You wont need to wait till the UI is opened and manually click on start.
Having Autostart set to true also allows for other Tools and Scripts to help you keep your servers running.
Getting help with Plugins
The bigger the server and older the map, the longer it takes to shut the server down and restart it. Sometimes it is even possible the server wont restart at all and be stuck. For that you can get some help from plugins.
One example is the ALE-RestartWatchdog (https://torchapi.net/plugins/item/370c6d2e-c2db-4eb2-a04f-49c9ee89dd2b)
After the server received the stop or restart command it waits for a specified amount of time (Default 120 Seconds) before forcefully shutting down the server and restarting it. Within these 120 seconds the normal saving process of the map will be completed. For unknown reasons stopping the machine causes the game to manually close each and every grid and voxel on the server which is basically the same as deleting it. It is unclear why, but it takes a long time.
So this plugin can speed up the whole restart. And in case your server will freeze while shutting down its one way to get it out of its misery.
Use external Watchdogs
There are Softwares which allow you to monitor your servers and the processes running on them. These however may be quite expensive and for your small server not really worth the effort.
In the end everyone can decide for himself if they want to invest into it or not.
However there are free and easy to set up alternatives using Windows Task Scheduler and a Batch file:
1 @echo off 2 SETLOCAL ENABLEDELAYEDEXPANSION 3 4 taskkill /f /fi "imagename eq Torch.Server.exe" /fi "status ne running" 5 taskkill /f /fi "imagename eq Torch.Server.exe" /fi "windowtitle eq Assertion Failed: Abort=Quit, Retry=Debug, Ignore=Continue" 6 taskkill /f /im werfault.exe 7 8 set "path=C:\Games\Space Engineers" 9 10 For /f "tokens=1-4 delims=/ " %%a in ('date /t') do (set mydate=%%c-%%b-%%a) 11 set "now=%mydate% %time%" 12 13 cd %path% 14 15 %SystemRoot%\System32\Wbem\wmic process get ExecutablePath > "%path%\runningProcesses.txt" 16 17 for /f "delims=" %%F in ('dir /B /AD "%path%"') Do ( 18 19 set "file=%path%\%%F\Torch.Server.exe" 20 21 if exist "!file!" ( 22 23 %SYSTEMROOT%\System32\find /c "!file!" "%path%\runningProcesses.txt" 24 IF ERRORLEVEL 1 ( 25 26 echo %now% !file! >> "%path%\autorestart.log" 27 start "" "!file!" 28 ) 29 ) 30 ) 31 32 del "%path%\runningProcesses.txt"
This file first checks for running Torch.Server.exe instances that are not considered running by Windows. So everything not responding will be killed.
Next Torch Instances with a certain error dialogue (usually occurring when you open torch twice) will be removed.
And the Windows Error Reporting Dialogue which may appear when an application is frozen will be terminated in order to get stuck torch instances shutdown.
Next this file will check for a folder structure located at "C:\Games\Space Engineers". This location you may need to alter on your system. The script will check All subfolders and look for Torch.Server.exe. If it does find one it checks if its already running and if its not it will be launched.
This is an example folder Structure with 4 Servers:
- C:\Games\Space Eningeers
- PVP Server
- PVE Server
- Hardcore Server
- Development Server
- PVP Server
So if for example the PVE Server is not running at the moment (so the process does not appear in task manager) it will be started and next to the batch file there will be added an autostart.log which tells you that it needed to be started up.
Using this technique however does not allow you to have a server shut down on purpose. So if you happen to do some bigger changes you need to deactivate the file first.
How to set up a Batch file with Windows Task Scheduler
To run your Batch file automatically you can set it up in the Windows Task Scheduler. Just add a new Task. Schedule it to run when the user Logs on, or the task is edited. And repeat every 5 Minutes.
The action will be "start a program" with your batch file selected.
If needed you can also run it with elevated rights, but it should not be necessary unless you run Torch as administrator.