Piloting an automated clinical trial eligibility surveillance and provider alert system based on artificial intelligence and standard data models

Three objectives guided the design of the pilot study of the TAES prototype. First, we aimed to implement an automated trial eligibility surveillance system that would extract and normalize clinical information from structured and unstructured EHR content and match it with normalized eligibility criteria from clinical trial protocols. Second, we wanted to develop a user interface for researchers to access the trial-matching database, select clinical trials, review the extracted eligibility criteria, define patient populations, and examine matching patient records along with the available evidence used to determine possible eligibility automatically. Finally, we wanted to assess options for connecting the trial eligibility surveillance system with a commercial EHR system for provider (and patient) notification.

As depicted in the top section of Fig. 1, the TAES prototype detects potentially eligible patients by acquiring trial eligibility criteria from protocols, extracting clinical information from the EHR, and identifying matches between the two sets of information.

Trial Eligibility Surveillance (TAES) system overview. DW = data warehouse; NLP = natural language processing

For abstraction of trial eligibility criteria, we selected five cardiovascular and cancer trials open for enrollment at MUSC with at least ten enrolled participants. All eligibility criteria, regardless of their likelihood to be extracted by an NLP system (as discussed in "Study limitations" below), were then retrieved from study protocols (or other sources) and manually abstracted using an electronic tool that enables domain experts to represent eligibility criteria in a structured and coded form (ATLAS open-source software tool, available from the Observational Health Data Sciences and Informatics (OHDSI) consortium [38]. We use the OMOP CDM with a selection of standard terminologies for representing these criteria, as this is a well-established model.

Clinical information stored in MUSC’s EHR is currently exported in real time or daily to an institutional clinical data warehouse. The TAES prototype extracts clinical information corresponding to eligibility criteria from EHR notes and represents this information with the same CDM and terminologies as the eligibility criteria in the trial-matching database. Extracted concepts can be grouped into six categories: conditions or diseases, investigations, medications, procedures, devices, and demographics. The “conditions and medications” extraction was enriched with six binary contextual attributes, which indicated if the extracted information was negated, uncertain, conditional, generic, historical, or not about the patient. Since a majority of the clinical information is recorded in narrative text only, we also used NLP to extract structured and coded information. We used DECOVRI (Data Extraction for COVID-19-Related Information) as the NLP tool for information extraction, a locally-developed and freely available open-source NLP tool built on Apache UIMA [39, 40]. DECOVRI was originally developed to extract COVID-19 related information, but it’s modules for extracting medications, demographics, and contextual attributes were considered sufficient for this task (i.e., good accuracy was measured with similar information extracted from clinical text notes in previous evaluations of DECOVRI, with gender and age extracted with 100% recall and medication attributes extracted with 68–98% recall). To adapt it to this new task, we added custom lexicons for conditions, procedures, investigations, and devices. These lexicons were generated with lex_gen, a freely available open-source tool that uses the UMLS Metathesaurus relations to create rich lexicons from a seed set of concepts [41].

Eligibility information is stored in the trial-matching database, along with all supporting data, for subsequent access. In this pilot study, we used the rule-based approach we experimented with earlier to assess trial eligibility [21]. This approach uses rules implemented as database queries exported from ATLAS and then applied in a database management tool. We first evaluated the available structured coded information relevant to trial eligibility criteria as a baseline. We then evaluated how the inclusion of information extracted by our NLP system improved or honed the review process for finding likely eligible patients. The queries were used to determine how many individual eligibility criteria a patient met for a given trial out of all possible criteria. The maximum score (e.g., 12 if there were 12 criteria) means all criteria are met, while a score of zero means no criteria are met.

Domain experts, including medical residents and advanced medical students with clinical documentation experience, built a reference standard based on the five selected trials to measure the accuracy of the automated patient eligibility assessment. We used historical enrollment decisions for a stratified random selection of 400 patients (including at least 100 enrolled in the selected trials) in the reference standard. The domain experts reviewed the EHR records of the selected patients and annotated information matching eligibility criteria using a secure web-based annotation tool (INCEpTION [42]). To guide their annotations, experts were provided with an annotation schema that matched the six general extraction categories described above. Annotators were also asked to flag any medication indicated as an allergen and annotate non-medication allergen mentions. For the detailed evaluation of the information extraction process, a random selection of 20 text notes from the aforementioned dataset was annotated in detail (i.e., all corresponding text spans and local context information). We then compared this reference standard with the extracted clinical information and the automatic eligibility classification to measure sensitivity, positive predictive value, and the area under the ROC curve (AUC).

The trial-matching database includes eligibility criteria, potentially eligible patients, and the patient information that matched eligibility criteria. The database is mostly based on the OMOP CDM, with the addition of custom tables dedicated to trial-patient matching information. Figure. 2 provides an overview of the database architecture, with links to the pre-defined tables in the OMOP CDM. The TRIAL table contains metadata (e.g., the name of the study’s principal investigator) that does not fit into the OMOP CDM’s COHORT table. The CRITERION table includes executable definitions for each eligibility criterion defined in a trial. For this pilot study, we embedded the database query code to extract all patients matching a specific criterion as the executable definition. The query result provides the rows of evidence to be added to the MATCH_EVIDENCE table. All individual instances of evidence for a patient, criterion, and trial triplet are aggregated into a single summary score in the MATCH_CRITERION table. Likewise, all individual criterion scores for a patient and trial pair are aggregated into a single summary score in the MATCH table. These scores help filter matches so that study teams can focus on the most promising ones.

Trial Eligibility Surveillance (TAES) database schema

A user-friendly web application provides access to trial-patient matching information, clinical trial search and selection, potentially eligible patients for further screening, and a visualization of matching patient records along with the available evidence used to a determine possible eligibility automatically (e.g., diagnostic or treatment code or information highlighted in the text note from which it was extracted). The web application was developed using the flexible Ruby on Rails platform with a Bootstrap [43] front end to simplify the user experience while providing a robust, elegant platform on which to build. The web application was designed to enable users to search, identify, and flag potentially eligible patients quickly. The selected matches could then be easily exported for further eligibility screening.

The OHDSI WebAPI enables interactions with the OMOP CDM database of extracted patient and trial information [44]. The OHDSI ATLAS platform provides access to the OMOP CDM database for detailed data exploration and population analysis, terminology browsing, cohort definition, and other database queries.

For this pilot study, members of the MUSC biomedical informatics and information systems teams considered a variety of options for integrating information from the TAES trial-matching database into the Epic EHR used at MUSC. They also considered options for communicating matches to healthcare providers (i.e., clinical workflow integration). Options were identified from electronic documentation and summarized into strategies and subsequent procedures. The potential strengths and weaknesses of each option were analyzed. Findings were then presented and discussed with an ad hoc trial eligibility notification stakeholders’ group that was created to guide integration efforts. The overarching aim was to explore possible options to integrate TAES with a commercial EHR system to then make providers aware of patient trial eligibility as early as possible during a clinical encounter, using documentation tools familiar to physicians so as not to disrupt workflow. In the future, patients interested in such notifications could also enroll through the EHR patient portal (e.g., MyChart for the Epic EHR system).