NICE releases guidance on integrating AI into real-world evidence generation

NICE has released guidance for use of AI in real-world evidence (RWE) generation, emphasizing transparency, rigorous validation and human oversight to ensure ethical, accurate health technology assessments.

The National Institute for Health and Care Excellence (NICE) has outlined a clear position on the use of AI in RWE generation, recognizing its potential to enhance health technology assessment (HTA) processes while also acknowledging the risks involved. As AI methods, including machine learning and generative AI, continue to advance, NICE anticipates their increasing role in evidence generation. However, the organization stresses that their use must be carefully managed to ensure transparency, accuracy, and alignment with existing ethical and regulatory standards.

AI is increasingly recognized for its ability to handle and analyze large datasets, which is crucial in RWE generation. NICE highlights that “AI methods may have a role to play in data processing before the development of real-world evidence”, such as transforming unstructured data into structured formats and integrating various data sources. These capabilities can significantly streamline data preparation, making RWE more accessible and actionable in HTAs.

The selection of relevant populations and data points from extensive datasets is critical for addressing specific research questions in HTA. AI’s ability to efficiently identify pertinent data ensures that the evidence generated is both relevant and robust. NICE acknowledges that AI can support the estimation of comparative treatment effects by employing advanced feature selection methods and leveraging the predictive capabilities of machine learning algorithms.

Despite these benefits, NICE emphasizes the need for a cautious approach, underscoring that AI methods “should only be used when there is demonstrable value from doing so”. The use of AI in RWE generation must be accompanied by rigorous validation processes to mitigate risks associated with algorithmic bias, transparency and cybersecurity. NICE requires transparency in reporting AI usage, including detailed documentation of the AI methods applied, their rationale and the steps taken to address potential risks. Organizations using AI in RWE generation are expected to adhere to established guidelines and checklists, such as the PALISADE and TRIPOD+AI frameworks, to ensure the accuracy and reliability of their findings. The organization advises that “submitting organizations should clearly justify the use of these methods and outline assumptions”, while ensuring that the application of AI does not compromise the quality or integrity of the evidence produced.

NICE also stresses the importance of human oversight in AI-driven processes, stating that AI should augment, not replace, human involvement. The principle of having “a capable and informed human in the loop” is central to maintaining trust in AI’s contributions to HTA. This approach aligns with broader ethical frameworks guiding AI development, ensuring that human judgment remains a cornerstone of decision-making in healthcare.