Troubleshooting DNS Resolution Failures on Remote-Agent (Linux)

7 min

Context: Running playbook jobs can either be on the $default or $remote pools. Jobs in the $default pool run directly on our swimlane-provided agent instance. When jobs are run in the $remote pool, they are executed via a remote-agent installed on the customer’s internal Linux (Ubuntu or RHEL) VM.

These remote-agents require reliable DNS resolution to maintain a heartbeat connection with the Turbine instance. If the DNS fails to resolve the turbine domain, then the remote-agent will get disconnected, resulting in failed jobs in the $remote pool.

This KB helps customers understand and troubleshoot such DNS resolution failures on their remote-agent servers.

Introduction: A DNS (Domain Name System) resolution failure occurs when your server is unable to convert a domain name into its corresponding IP address. This typically happens due to issues with your internal DNS configuration, network connectivity, or the upstream DNS server itself.

When this failure occurs on a remote-agent server, it loses connection to our instance. This leads to job failures for any workflows scheduled to run in the $remote pool.

Symptoms:

Jobs fail in the $remote pool with connection errors

Remote-agent appears disconnected in the UI

DNS resolution fails when tested with tools like dig or ping

What to Check When the Issue Happens:

Basic DNS Testing Run the following to test DNS from your remote-agent: (Replace the domains and IP addresses below as required)

dig @127.0.0.53 xyz.com dig @1.1.1.1 example.com ping test-instance.com

Check System DNS Resolver

systemctl status systemd-resolved journalctl -u systemd-resolved

Review DNS Configuration Check /etc/resolv.conf to see what nameservers are configured.

Capture Logs During the Incident Please collect the following logs when the issue occurs: (Replace the example.com with domain name of turbine instance)

cat /etc/resolv.conf dig example.com ping example.com curl -v example.com systemctl status systemd-resolved journalctl -u systemd-resolved --since "10 minutes ago"

Post-Incident Checks:

After the system starts working again, gather the same outputs to compare: (Replace the example.com with the domain name of turbine instance)

dig example.com cat /etc/resolv.conf

You may also want to capture routing and network status:

ip route ip addr nmcli device show

Monitor DNS Resolution Automatically:

Set up a cron job to run hourly and log DNS query times for monitoring.

Script (Save as /usr/local/bin/dns_monitor.sh): (Replace example.com with the domain of your turbine instance)

#!/bin/bash timestamp=$(date "+%Y-%m-%d %H:%M:%S") log_file="/var/log/dns_monitor.log" dig_local=$(dig @127.0.0.53 example.com +stats | grep "Query time") dig_public=$(dig @1.1.1.1 example.com +stats | grep "Query time") echo "[$timestamp] Local DNS: $dig_local" >> $log_file echo "[$timestamp] Public DNS: $dig_public" >> $log_file

Make script executable:

chmod +x /usr/local/bin/dns_monitor.sh

Add to Crontab:

crontab -e Add this line:0 * * * * /usr/local/bin/dns_monitor.sh

Additional Network Troubleshooting:

If DNS appears fine but issues persist, try the following:

Check network interfaces:

ip link show

Test external connectivity:

curl -v https://example.com traceroute example.com

System logs:

journalctl -xe dmesg | grep -i network

NetworkManager logs (if used):

journalctl -u NetworkManager

Next Steps:

If the issue recurs, please collect and share the logs above when opening a support ticket.

Your internal network team may also assist in reviewing DNS or firewall configurations.

If you need help analyzing the logs or confirming root cause, feel free to reach out to our support team. We're happy to assist.