@WickedShell and I were talking about the whole CI situation before him making the proposal, so he basically knows my thoughts, but here they go so everyone (and specially the funding committee) know.
I agree with Michael that CI is taking too much time currently. In the last months we added Sub, px4-v3 and px4-v4pro, which together have all impacted Travis in a big way; mainly the last two, because, just like all other px4-v* jobs, they are doing two builds, each one using one of the two builds systems we still support.
When we stop doing builds with the Make build system in Travis, runtime of five jobs will be cut by half (or more). It will certainly help a lot, but I'm not sure if it will be enough - it's the kind of change that can only be tested on production, with PRs being opened, updated and merged. We have now switched our firmware server to build everything using waf and I think this is a good time to change CI too. For a long time, Linux boards and SITL have been built in CI just with waf, with a Cron job running once a day to make sure the Make build system still works - that has lately timed out in Travis too.
Regarding Semaphore, I have two issues with it:
- its cache feature is almost useless, because it is shared by all parallel jobs (meaning we can't cache ccache database for example)
- its configuration is done on their website instead of using a file in our repo like Travis and others do
I'll explain why the second one is important: just like ArduPilot has free Travis and Semaphore accounts, everyone can have them too. This means that everyone can put their code to the test before even opening a PR. The good part about Travis is that anyone using master as a base will run exactly the same test that we do when a PR is open. Semaphore doesn't allow that - if we change the Semaphore configuration and a user doesn't know about it, their personal Semaphore account will run with any older configuration the user has put there.
I've started reading CircleCI documentation, but with the amount of work I have daily, I can't give any promises of a deadline to make it work.
It pains me a bit to put this amount of money to get just two more jobs, but if the team wants to, I'll certainly work to put them to good use.