Machine learning and artificial intelligence seem to have dipped into almost every industry in the world; fashion, marketing, manufacturing, engineering and even commercial real estate. So, while it’s not hard to fathom that machine learning could soon leak into the pharmacovigilance and safety monitoring arenas, it is difficult to predict where these technologies can make drug safety more efficient without disrupting it’s current flow.

Study Background

A recent study that finished in late December of 2023 published the results of using machine learning models to detect safety signals within FDA databases. The purpose of the study was to determine if artificial intelligence was more capable of pinpointing variables that could potentially offset or mask each other than traditional signal detection methods. 

Before the study began, a machine learning model known as ‘Logistic Regression’ had shown success in predicting offsetting and masking variables, and was used as the first of the four comparison models. However, logistic regression does have some rather large limitations, and called for the development of more complete machine models. 

The four models are as follows: 

Logistic Regression

• Originally studied due to its ability to identify confounders and masking factors more efficiently than traditional methods 
• Was found to have challenges when working with pre-programmed interactions, large databases, outlier observations, non-linear relationships and correlated independent variables 

Random Forest

• Originally created to predict adverse drug effects from in vitro profiles utilizing in vitro pharmacological assay data from FAERS
• Exemplified high accuracy, precision and recall data

Gradient-Boosted Trees & Support Vector

• First used to predict hospitalizations and deaths based on patient demographics and drug usage 
• These models were 73%-75% accurate when predicting hospitalization, and 68%-76% accurate when predicting deaths
• Both models have high recall, F1 and precision scores

Methodology

Using the FDA adverse event reporting system known as FAERS, researchers gathered data ranging from October 1st, 2017 to December 31st, 2020. 

Results were measured using both a crude model and a synthetic minority oversampling technique (SMOTE), and because the data gathered was slightly unbalanced, two training sets were created to achieve non-biased study results: 

Set 1: Stratified on the outcome variable 

Set 2: Synthetic minority oversampling technique (SMOTE)

Lastly, model performance was compared using the following criteria:

• Accuracy 
• Precision 
• F1 score 
• Recall 
• Receiver operating characteristic area under the curve (ROCAUC)
• Precision-recall curve area under the curve (PRCAUC)

Results & Conclusions

After comparing the machine models, researchers found that all four methods yielded better accuracy, F1 and recall results when using the stratified on the outcome variable set rather than the SMOTE technique. 

Taking a closer look, logistic regression still had the highest accuracy, precision and recall results when compared to the other three models. That being said, gradient-boosted trees showed the highest ROCAUC result at 0.646 and random forest produced the highest PRCAUC result at 0.839. Ultimately, all methods had similar F1 scores, a PRCAUC greater than or equal to 0.8 and a ROCAUC greater than or equal to 0.5. 

This study was a success in terms of finding additional machine models that had similar capabilities and less limitations than logistic regression. Additionally, all four models showed the ability to produce additional case information that resulted in researchers identifying more signals than what was expected. 

Altogether…

There is still so much to be discovered within the application of any type of artificial intelligence to pharmacovigilance practices. While most industry professionals agree that humans should always have control over each step of the safety signal detection process, the question still remains: 

Should humans be solely responsible for every detail of this process or is there room for the integration of machine learning within our safety monitoring practices? 

Source: Evaluation of Four Machine Learning Models for Signal Detection