Medical artificial intelligence (AI) has enormous potential to transform medicine. Over 521 medical devices enabled by AI or machine learning have been approved for use in the USA. Yet, according to labour trends, healthcare lags noticeably behind other industries in AI adoption. One reason is that we cannot currently measure the trustworthiness of AI systems in healthcare, including more-recent generative AI tools. To measure trust in medical AI tools, we should score the explanations that they generate against the reasoning provided by medical experts. This process must include transparency of training data and open community vetting.