Recent addons commit meant that addons would be enabled even if they weren't found.
This would give an error (which is fine), but also remove from preferences.
Too many contrib addons are in an unstable state making the test not so useful.
Thanks to Sergey initial patch: D1012, redid mostly - but outcome is the same.