This community is in archive. Visit community.xprize.org for the current XPRIZE Community.
Pilot Testing & Evaluation Frameworks for Digital Health Solutions
TerryMulligan
Posts: 38 XPRIZE
in Prize Idea
Testing is a major component of an XPRIZE, and understanding the current best practices of digital health solution testing is essential in order to identify which parameters should be applied for this competition.
• What were your experiences with piloting digital health solutions in low-resource environments?
• What were your criterion for evaluation and success?
• What frameworks can be used to evaluate a digital health program?
• What were your experiences with piloting digital health solutions in low-resource environments?
• What were your criterion for evaluation and success?
• What frameworks can be used to evaluate a digital health program?
0
Comments
Hi @Davisthedoc, @Debbie_Rogers and @aassif_lg - As you all have successfully designed and implemented a Digital Health Solution programs - Please share your experience on a framework used to evaluate such a program / service.
Impact Evaluation is critical as well, and we use as part of a closed-loop process of continuous improvement. We use a tiered outcome model to assess the impact of telementoring for healthcare and/or other professionals. These levels are common across multiple training/education evaluation frameworks like Moore's 7-level outcome model and Kirkpatrick's model.
- First we make sure people participate regularly, engage, and feel like they get something out of the program.
- Then we see if they learned something or are more confident in their knowledge
- Do they intend to use that knowledge to change their behaviors, or ideally, did they actually change some behaviors or practices?
- Ultimately, we want to see if these changes in practice have an impact on patients. Are more patients treated? Do patients spend less time/money and miss less work to get treated? Do patients get better treatments? Do measures of disease severity improve? Are patients more satisfied with their care?
- Finally, we want to see if any of these changes go beyond individual practitioners or patients. For example, do rates of disease burden go down in healthcare facilities or communities where a lot of practitioners participate in ECHO? Is there evidence that best practices are adopted by organizations or systems such as public insurance?
After a specific intervention like ECHO has been shown to be effective, we also use other frameworks to evaluate how well it is being scaled such as the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. This framework helps us think about why certain entities or practitioners do or do not adopt ECHO, and what factors need to be in place for an ECHO program to be sustainable. These frameworks help us assess the public health impact of an intervention. For example, you could have a really effective intervention but it's either difficult to access or very expensive, so it doesn't scale well. The goal is to have the most effective intervention that ALSO can scale at the level of having a public health impact.Also, we completely agree "that the goal is to have the most effective intervention that ALSO can scale at the level of having a public health impact." Thank you so much for your contribution.
I also think it depends on what the interventions are and who the target population is. Curing diseases vs. promoting health and wellness through digital health solutions? Children, mothers, older people, and/or all of them? I am not sure whether or not the XPrize team has already decided those specific features of the competition.
Effectiveness, process, and cost evaluations are often planned and written in most study protocols for digital health solutions, although not all studies cannot keep the promises.
Digital literacy can or should be considered as an outcome again depending on the target population and programs.
Let me know if you have any questions and hope to see you on Friday.
We are using Vanilla platform for hosting this community.