Autotest is losing old tlog files

Old tlog files are getting lost and replaced by PNG files.

If someone wants to check, just try to download any tlog that is older than November.
I have also created a script to check each tlog file available.

#!/usr/bin/env python3
import magic
import posixpath
import requests

from bs4 import BeautifulSoup

ARDUPILOT_AUTOTEST_HISTORY = "https://autotest.ardupilot.org/history/"

def get_all_folders_urls():
    page = requests.get(ARDUPILOT_AUTOTEST_HISTORY)
    soup = BeautifulSoup(page.content, 'html.parser')
    dates_href = [date['href'] for date in soup.find_all('a', href=True) if '2021' in date['href']]
    return [posixpath.join(ARDUPILOT_AUTOTEST_HISTORY, date_href) for date_href in dates_href]

def get_all_tlogs_urls(folder_url):
    page = requests.get(folder_url)
    soup = BeautifulSoup(page.content, 'html.parser')
    artifacts_href = [artifact['href'] for artifact in soup.find_all('a', href=True) if artifact['href'].endswith('.tlog')]
    return [posixpath.join(folder_url, artifact_href) for artifact_href in artifacts_href]

for folder in get_all_folders_urls():
    for tlog_url in get_all_tlogs_urls(folder):
        data = requests.get(tlog_url)
        print(f"{tlog_url} is a {magic.from_buffer(data.content)}")
https://autotest.ardupilot.org/history/./2021-05-01-07:05/HeliCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                       
https://autotest.ardupilot.org/history/./2021-05-01-07:05/Rover-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                            
https://autotest.ardupilot.org/history/./2021-05-01-07:05/ArduPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                        
https://autotest.ardupilot.org/history/./2021-05-01-07:05/APMrover2-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                        
https://autotest.ardupilot.org/history/./2021-05-01-07:05/Soaring-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                          
https://autotest.ardupilot.org/history/./2021-05-01-07:05/ArduSub-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                          
https://autotest.ardupilot.org/history/./2021-05-01-07:05/QuadPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                        
https://autotest.ardupilot.org/history/./2021-05-01-07:05/BalanceBot-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                       
https://autotest.ardupilot.org/history/./2021-05-01-07:05/AntennaTracker-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                   
https://autotest.ardupilot.org/history/./2021-05-01-07:05/ArduCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                       
https://autotest.ardupilot.org/history/./2021-05-01-16:05/HeliCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                       
https://autotest.ardupilot.org/history/./2021-05-01-16:05/Rover-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced                                            
https://autotest.ardupilot.org/history/./2021-05-01-16:05/ArduPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/APMrover2-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/Soaring-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/ArduSub-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/QuadPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/BalanceBot-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/AntennaTracker-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-01-16:05/ArduCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/HeliCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/Rover-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/ArduPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/APMrover2-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/Soaring-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/ArduSub-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/QuadPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/BalanceBot-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/AntennaTracker-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-00:05/ArduCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/HeliCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/Rover-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/ArduPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/APMrover2-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/Soaring-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/ArduSub-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/QuadPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/BalanceBot-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/AntennaTracker-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-02-08:05/ArduCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/HeliCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/Rover-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/ArduPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/APMrover2-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/Soaring-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/ArduSub-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/QuadPlane-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/BalanceBot-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/AntennaTracker-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-03-00:05/ArduCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-05-01:05/HeliCopter-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
https://autotest.ardupilot.org/history/./2021-05-05-01:05/Rover-test.tlog is a PNG image data, 600 x 600, 8-bit/color RGB, non-interlaced
...

Those tlog files are actually simply not present on the autotest server - removed by a cleanup cronjob, I believe.

The Apache “MultiViews” module is assuming that because (e.g.) HeliCopter-test.tlog doesn’t exist you would really like HeliCopter-test.tlog.png instead!

We’ll discuss cleaning the .png files up at the same time we do the tlog

Hey @peterbarker, any update ?