25.1.2 to 25.1.9 Troubleshooting Remote Agent Unavailable

5 min

If a Remote Agents becomes unavailable, and the connections with TC are not reestablished until the Remote Agent container is restarted.

You can use the following to handle the below error:

Add a new cron job with the user who runs the containers to restart automatically the remote agent when we detect the "Unexpected error occurred during agent heartbeat" error message.

NOTE: In case you would like to apply this script, you need to edit the script and change the "AGENT_CONTAINER_NAME" value

Open the script corresponding to the container technology you are using, Docker or Podman. (Scripts attached)

Edit the script > add the turbine agent name > Save

How to get the remote agent name? Run the following to list the containers running

docker/podman ps

Give execute permissions to the scripte.i (Change the file name to the correct script name)

chmod +rwx remote-agent-monitoring-(docker/podman).sh

Edit crontab

crontab -e

Add a cron job (Change the all path if it's necessary, and the script name)

*/5 * * * * /home/<user-name>/remote-agent-monitoring.sh >> /home/<user-name>/remote-script.log 2>&1

Save

*press esc* > :wq

Code:

Docker script:

#! /usr/bin/env bash DOCKER_BIN=$(which docker) #DOCKER_BIN="/usr/bin/cat" AGENT_CONTAINER_NAME="<replace_with_your_agent_container_name>" AGENT_LOGS=$(${DOCKER_BIN} logs --tail 20 "$AGENT_CONTAINER_NAME" 2>&1 | grep "Unexpected error occurred during agent heartbeat") #AGENT_LOGS=$(/usr/bin/cat logs.txt 2>&1 | grep "Unexpected error occurred during agent heartbeat") length=$(echo "$AGENT_LOGS" | wc -l) echo $length if [[ ! -z "${AGENT_LOGS}" && ${length} -gt 5 ]]; then echo "- Restart Agent: $DOCKER_BIN restart $AGENT_CONTAINER_NAME" AGENT_RESTART=$(${DOCKER_BIN} restart "$AGENT_CONTAINER_NAME") fi

Podman script:

#! /usr/bin/env bash PODMAN_BIN=$(which podman) #PODMAN_BIN="/usr/bin/cat" AGENT_CONTAINER_NAME="<replace_with_your_agent_container_name>" AGENT_LOGS=$(${PODMAN_BIN} logs --tail 20 "$AGENT_CONTAINER_NAME" 2>&1 | grep "Unexpected error occurred during agent heartbeat") #AGENT_LOGS=$(/usr/bin/cat logs.txt 2>&1 | grep "Unexpected error occurred during agent heartbeat") length=$(echo "$AGENT_LOGS" | wc -l)echo $length if [[ ! -z "${AGENT_LOGS}" && ${length} -gt 5 ]]; then echo "- Restart Agent: $PODMAN_BIN restart $AGENT_CONTAINER_NAME" AGENT_RESTART=$(${PODMAN_BIN} restart "$AGENT_CONTAINER_NAME") fi