58 – Expanding the ORCA Measure

Learn more about ORCA in this blog post.


0:08i’m so happy that you’re here today we’re excited  to be having a workshop to describe what we’re  

0:17doing with the orca observer reported  communications abilities measure that  

0:26we’re working on at combined brain with the crack  team at duke university and it is really by the  

0:35way a pleasure to work with the duke team i i  just it’s really fun so we also have attending  

0:43this workshop representatives from several of the  uh patient advocacy groups who are participating  

0:52and we may have them be popping up and asking  or answering questions but for right now  

0:59i would just like to introduce christy ziegler  and leslie zapata leyva which i probably pronounce  

1:06your last name wrongly leslie and so i’ll start  with christy if you can introduce yourself and  

1:16you know what i forgot to introduce myself this  is terry jo michelle i am the director of combined  

1:23brain i am a neuroscientist i am specialized in  biomarkers and outcome measures as is combined  

1:31brain which is the consortium for outcome measures  and biomarkers for neurodevelopmental disorders  

1:38and we represent 35 different neurodevelopmental  disorders all trying to fast-track clinical trials  

1:45and with that i would like to introduce christy  and then you can introduce leslie and we’ll take   it from there thank you yeah thanks so much  and i just wanted to say it’s such a pleasure  

1:56to work with you all as well and we’re really  happy to be here today to talk with everybody   about our project and a little bit about what we  do and and why health measurement is so important  

2:07so i’m christy ziegler i’m an assistant professor  at duke i’m one of the core faculty members in   the center for health measurement so my training  was in research methods so the study of research  

2:18with a focus on statistics as well as  psychometrics which is the study of testing  

2:24and measurement so all of the work that i  do right now is developing and evaluating  

2:30outcome measures as well as using outcome measures  from a patient and family-centered perspective  

2:36and i focus specifically on rare diseases and  pediatrics so leslie i’ll turn it over to you

2:45yeah so i’m leslie i’m a clinical  research specialist at the center   of health measurement with christie and  i’ve been here for a little over a year now  

2:55officially and yeah it’s been a delight being a  part of this group and working on this project

3:03okay so can you tell us a little bit about the  project and what is what is it what’s it all about  

3:14yeah i hope it’s okay to share my screen i have  some some slides so we wanted to talk a little  

3:21bit about you know what we do with this who we are  at the center for health measurement what we do  

3:28and why outcome measures are so  important and why we focus on those   then i was hoping to get a little bit into the  development of the orca measure sort of where did  

3:37it come from what is its story and and what we’re  working on right now with the fda funded project  

3:42then leslie’s going to talk a little  bit about what families can expect   from participation and then we will wrap it  up with you know questions answers as well  

3:51as talking about like what’s our end goal  for this project if that sounds all right

3:58perfect all right before we jump in i have some  disclosures so it’s important to note that our  

4:03study team has developed the orca measure and if  it’s commercially successful in the future we may   benefit financially along with duke university  i’m also going to be talking about an FDA study  

4:14as part of this talk today so there’s some details  about that grant award right there on the slide  

4:20it also is really important to note that any  views expressed by me or leslie are our own  

4:25and don’t necessarily represent our funders  or our employer so let’s let’s jump right in  Center for Health Measurement

4:32so what is the center for health measurement  so we sit within duke university in the   school of medicine within the department of  population health sciences right now i think  

4:42our account is at 11 different faculty members  10 staff members and two postdoctoral trainees  

4:50and really what we’re focused on doing is  centering patients and caregivers to improve  

4:56their care their quality of life and so we really  do that in four big domains the first is really to  

5:04identify outcomes that matter right so what should  we be researching what should we be focused on  

5:10you know what should therapeutics be targeting  right like what matter to people who experience  

5:15different aspects of their health so stakeholders  could include patients parents caregivers but also  

5:22sometimes we engage with providers so clinicians  payers and regulators the second area that is  

5:28most relevant to this project is developing and  evaluating clinical outcome assessments and so  

5:34clinical outcome assessments or coas is just a  fancy word for survey tools so we’re going to talk  

5:39a little bit more about like what this looks like  what’s this typical process and why it’s important   we also have a group of folks that look  specifically at preferences for patients  

5:48and caregivers this is really aligned  with like decision-making processes   so in the clinical trial setting you can think  about this as them they’re trying to figure out  

5:58how people make decisions about taking medicines  or when do benefits outweigh risks or side effects  

6:04of a certain medication and then we also focus on  researching the research so what are the methods  

6:11that we use to do all these things how can we  do it better how can we do it more efficiently   how can we do it in a way that really does bring  that patient and family voice into the center of  

6:21research and clinical care so that’s really the  value i think of what brought us all together  

6:27at the center for health measurement is really  this idea of how do we bring people’s voices  

6:33into care planning treatment decisions  the health policy that really impacts   their day-to-day life right so the way that  we do this is through this measurement model  Clinical Trial

6:43and i’m going to give you an example uh within  the clinical trial setting because that’s really   what the project that we’re talking about today  focuses on so in a clinical trial right especially  

6:53with this this newer paradigm of patient-focused  drug development we typically have a medication  

6:59we’ve we know something about the safety of this  medication but our question is it does it work is  

7:05it working in the setting the way that we expect  it to so traditionally how we do that is we take   this medication we give it to a large group  of people then we monitor their health status  

7:16now when we’re evaluating if this measure or if  this medication is working or not we’re really  

7:25interested in one of three things how the person  is feeling how they’re functioning so can they go  

7:30out and live their life in the way that they would  like to and how they’re surviving now survival  

7:37as you can you know as if you’re thinking  about it it’s a little bit more objective   right it’s pretty clear from you know from their  medical chart if somebody is living longer or not  

7:48but if we’re interested in how somebody is  feeling or how they’re functioning right feeling   is something like symptoms pain fatigue this is  something happening inside somebody’s body fun  

7:59functioning is something that’s happening in  their life right that happens most of the time   outside of the clinic setting we really need  to talk to the person or their family their  

8:08caregivers to really get an idea of has their  life improved has their health improved within  

8:14this setting with this with this drug or this  medication and so the way that we do this in  

8:20a standardized way is we use a clinical outcome  assessment or a survey tool to get standardized  

8:26information from patients and their families  about how they’re feeling for their functioning  

8:32now within a clinical trial we’re making pretty  high stakes decisions right does this medication  

8:39work does it not work how well is it working and  so the way that we’re getting this information  High Quality

8:44about how a person is doing really depends a lot  on how we ask this question and what questions  

8:50we’re asking so our work is really focused in  on this clinical outcome assessment or a coa or  

8:57the survey to really identify if it’s high quality  now what does high quality mean when we’re talking  

9:03about a clinical outcome assessment we want it  to be meaningful it needs to matter to people  

9:09it needs to matter to their families it  needs to matter to these other stakeholders   and if it’s not meaningful the danger or the risk  that we run into is that we end up with drugs that  

9:18do something but it’s not the something that  people want to happen in their life right it’s  

9:23not the thing that’s going to really impact and  improve their quality of life we also want our  

9:30measurement to be valid this means that we’re  measuring what we want to measure so if we say   we’re measuring fatigue we’re measuring fatigue we  say we’re measuring communication we’re actually  

9:39measuring communication this also means that we’re  measuring the important components of that thing  

9:45so for communication for example this means that  we’re measuring all of the important pieces of  

9:51communication from the perspective of the people  that are being impacted by the clinical trial  

9:57we also want the measure to be reliable so  we want to get consistent stable information  

10:04we also want it to be sensitive so we want to make  sure that if a change is happening so we give a   medication person gets better we want to make sure  that the scores show that now if we end up with  

10:14a measure that’s not valid reliable or sensitive  the danger there is i would say probably the worst  

10:20case scenario is we have drugs that work but we’re  not able to show that they work within a clinical  

10:26trial right so we end up with clinical trials  that fail not because the medication isn’t doing   what we think it’s supposed to do but because we  weren’t able to show that with the measurement  

10:36tools so that is why i think the focus around  outcome assessments outcome development there’s  

10:44a lot of energy there’s a lot of focus there  especially in a clinical trial right because any  

10:52uh outcome assessment endpoint model goes in with  the drug you know proposal to the FDA so there has  

10:59to be buy-in from all sorts of stakeholders  right like pharma cares about measurement   the fda cares about measurement the patients  and the families care about measurement  

11:08so we want to make sure that we’re doing this the  right way so how do we know if a clinical outcome  

11:15assessment or a survey is of high quality  so i’m going to use the observer reported  Example

11:21communication ability or the orca measure  in our current project to give an example  

11:27i wanted to pause there see if there are  any questions on the initial initial piece

11:33does anybody have any questions about that i do can you go back to the slide that had  valid reliable okay so when you’re talking about  Valid vs Reliable

11:49a cl uh uh a coa uh i think we should probably  keep using the words for that instead of the  

12:00acronym so a clinical outcomes assessment  which for the most part is usually a survey  

12:07so when you’re talking about a survey and you’re  saying is it meaningful you actually want to know  

12:16if the questions you’re asking are something that  people want to tell you isn’t that what that means  

12:24exactly oh yes so you can think about it at the  broad topic level like should we even be measuring  

12:30this thing like fatigue or communication but also  in the way that we ask the questions right like  

12:36are these the components of communication  that matter to your family that represent   your experience so it’s sort of like broadly are  we measuring what we should be measuring but also  

12:48are we measuring it in a way that’s meaningful and  resonates with with families okay so now i’m going  

12:54to skip valid for a second and go to reliable  so just to restate that when you’re saying is  

13:00it reliable you mean if you ask me about my son’s  abilities today and you ask me again in two months  

13:12that i’m going to still give you pretty much the  same answers or do you mean that if you ask me  

13:19and then you ask my husband if both of us are  going to answer the same way what do you mean  

13:26yeah so reliable i think means both things so  we want to make sure we’re getting consistent  

13:31information we’re getting stable information  so you can think about that in terms of  

13:36people right so is it reliable when we talk to  you is it reliable when you talk to your husband  

13:42are we getting generally the same answers  and we can also think about it over time so  

13:48if we ask you in the morning and we ask you in  the afternoon but your son’s communication hasn’t   changed significantly we would want to see scores  that are really similar now i think over time it  

14:00we’re usually in the data we’re usually looking  to see closer scores closer responses if it’s the  

14:08same respondent but obviously you and your husband  have slightly different perspectives of your son   you interact with him in different ways so we  might see some differences we don’t want to see  

14:18really large differences right we want consistent  stable information over people over occasionsSensitive

14:25okay so now i want to still ask more  questions but i want to go to sensitive  

14:31okay so and then i’m going to take valid  last so for sensitive you’re saying  

14:38if something changes and that would be for better  or for worse can the survey pick up the change  

14:47so let’s say that my son he he doesn’t have words  per se he does have verbal approximations but  

14:58he does have a bunch of signs and he  communicates in all different ways   what if i felt like there was something new that  he did that i thought was kind of a big deal like  

15:11maybe instead maybe when he was hungry for lunch  he got a plate and brought it over to me and  

15:18i felt like that was communicating that he was  hungry for lunch so what you’re saying is that it  

15:26will this survey be sensitive enough to pick up a  change in something that i think is important and  

15:33meaningful in my life right yes yep okay so that’s  exactly right and i think when we talk about how  

15:42do we do that right how do we know that our or how  how are we ensuring or supporting the fact that uh  

15:50any measure is going to pick up on those things  that are meaningful meaningful changes both  

15:55improving but also worsening right we ground that  in actually talking to caregivers right because i  

16:02don’t know necessarily what’s meaningful to you  and you don’t know what would be meaningful to  

16:07me so we have to talk to people to really pick  up on what are those things that are meaningful  

16:12and how do we ask about them in a way that’s  going to pick up on those maybe subtle changes   that are really important so for one family that  might not be a meaningful change but for another  

16:25family that might be a meaningful change so you’re  trying to figure out what are the exact gradations  

16:32and what is a meaningful change yeah and i think  one of the things that we’ve seen is depending on  

16:38where your child’s current level of functioning is  what is meaningful improvement might look a little  

16:44different right depending on where you fall on  the spectrum so i think any measure that sensitive  

16:52has has got to show or include different ways for  meaningful improvement especially in communication  

17:00has to have different areas where you could see  growth so if a child for example really had a  

17:07hard time moving and maybe you know could couldn’t  use signs or or assistive devices but was able to  

17:19look at something that he really wanted  in an in a consistent way would that be  

17:26something that you could try to pick  up with the orca or is that going to be   kind of too hard to to notice that change  yeah that’s a great question so we we  

17:39part of our qualitative work in Angelman syndrome  which i have some slides on in just a minute  

17:45you know the there’s the modality or how  somebody is communicating mattered a lot so  

17:52the orca was designed to include those different  ways somebody communicates whether that’s  

17:58you know through sounds through eye gaze through  moving their body going to get something and  

18:03bringing it to the parents and also devices  whether they’re high tech or low tech like a  

18:08picture book or a talker or something like that so  we wanted to make sure we were not focusing just  

18:14on verbal speech like some of the existing  measures really do that we were broadening  

18:19the way that we think about communication and how  individuals communicate so that we can get at that  

18:25those important behaviors yeah so okay so now i  want to get at the valid question okay great okay  Validity

18:35so whenever we talk about the orca we say we’re  trying to validate it and a whole bunch of just  

18:43different disorders that it was um started for uh  angelman syndrome but now we’re trying to validate  

18:52it so you explained valid a second ago but i just  want you to break it down into a smaller piece  

18:57what what does that mean yes so at a minimum  or a broadly broadly a broad umbrella term  

19:09validity means or valid means that we’re  measuring what we intend to measure so   when we say we’re measuring communication  we actually are measuring communication  

19:19now there also are different ways that we  support the idea of something being valid  

19:26so we gather a lot of evidence and the way  that we do that is through multiple different  

19:34research methods so we do qualitative work so  that’s interviews with caregivers to help us  

19:40define what communication ability is what it  looks like what are the important components  

19:45we do testing of a measure which where we sit down  with parents and we give them a draft measure and  

19:50we say like hey how is this working what questions  do you have is your experience reflected in these  

19:56questions like what should we change so that  all helps us to build this supporting evidence  

20:02for a measure to the validity of the  measure we also gather quantitative data  

20:08um so we give the measure to a large a  large relatively large group of parents and  

20:14and then do some statistical tests on it to  help to also help us to support the validity  

20:20of a measure or not so i think validity to  me is probably the biggest bucket term and  

20:29reliability and sensitivity could perhaps fall  underneath the bucket of validity right but at  

20:36a minimum are we measuring what we want to measure  so if it weren’t val if it was not a valid survey  

20:45uh then you might what would what  what does that mean it’s not yeah  

20:52well i think the danger is is that you don’t  see changes that are really happening because  

20:59you’re not asking the right questions and  so an example of that in communication   we want to measure communication so we pull a  measure off the shelf and it’s asking us about  

21:09vocabulary grammar pronunciation right so we ask  these questions it’s communication right but it’s  

21:16not the content of the measure isn’t matching  the experience of how people are communicating  

21:23so if we don’t see any changes in that way that  we’re measuring communication but your but your  

21:29son is going to get the plate and bringing it to  you and he’s hungry right that’s a major mismatch  

21:34in content so we’re not actually measuring  the full picture of communication ability if  

21:40we’re just asking those subset of questions  so i think validity is it right we need to  

21:46have support for validity to use a measure and if  we’re thinking about content is the content right  

21:52then the way that we have to establish  the contents right is to talk to parents  

21:58and see what what that concept means to them  and how does it show up in their life what do   they notice what’s important what really what new  skills has their child gain that really did help  

22:07you know like they can request what their favorite  food for breakfast right and that improves  

22:13um that interaction so i think validity to me is  olive is the thing that we’re trying to get at  

22:20and if we’re missing the mark on validity  we really run the risk of not being able to  

22:27trust scores that we get from the measure okay  thanks uh anybody else have any questions or  

22:35should we let christy keep going with  the next part of the presentationChallenges

22:41okay christy unless somebody pops something oh i  do have a question can you address the challenges  

22:48uh jayada do you wanna go ahead and turn your  um camera on and ask that question no you can  

22:54just read it okay she says can you address the  challenges of identifying measures that are  

23:01meaningful valid reliable and sensitive which can  also be observed in the limited time of a trial  

23:09yes yes so this um it sounds like a question about

23:17maybe the limit so i think there’s two  things that this question brings to mind   number one is administrative burden so how long  does it take to give these measures to people  

23:28how long does it take to gather the  information within a clinical trial   and that’s something that we’re always thinking  about especially if we’re developing a new measure  

23:36like the orca where we’re we’re intending for it  to be used in clinical trials we’re thinking about  

23:41how it’s administered how long it takes you know  how often we should be giving it to parents right  

23:48i think the second question or the second thing  that your question brings up is also how do  

23:54we do all of this validation work when these  clinical trials are coming down the pipeline  

24:01a lot quicker than you know maybe we would like  five years to do all this fancy measurement  

24:07deep dive right but we a clinical trial is coming  next year or in six months or in two months so how  

24:12do we how do we use something that’s reasonable  enough that we’re confident that it’s going to  

24:18pick up on changes if they exist i think those  are two different um questions but those are  

24:23absolutely considerations when we’re developing  or we’re pulling a measure off the shelf rightWhat can be measured

24:32christy if i could just uh clarify i think  it’s more the point of my question was more  

24:38about for severely affected kids  how much can meaningfully change  

24:45in that limited time period so you’re working  on communications but all these other things  

24:52i mean my grandson’s been working on head  control for nine years and um and he makes  

24:59meaningful improvement but what could be  measured you know in a 10 16 week period  

25:08that would really reliably capture progress  yeah that’s a great question so i think that  

25:18the answer to that is it depends and it depends  on the concept that you’re that you’re measuring  

25:25and the therapeutic or the anticipated  mechanism of action of the drug so how is  

25:30the drug supposed to work and what changes are we  expecting to see within that window right and um  

25:39it really depends on so a measure that picks up  on broad changes really reliably might not be a  

25:49good measure to plug into a clinical trial that’s  only 10 or 12 weeks right where you’re looking at  

25:55you need to see improvements that might  be neat you might need something much more  

26:00sensitive much more gradient for that 10 to  12 weeks so you can pick up on those emerging  

26:06emerging skills right so i think it really  depends on how you’re using the measure  

26:13and what time frame you’re focused on that  helps you pick what’s a high quality measure  

26:20and it might look different depending on  the clinical trial right and i guess also uh  

26:26what we should say is the orca may not be  the measure that every single person can use  

26:33at every age and every severity severity level  that it may be that after the orca studies are  

26:43done we know that it’s meaningful valid reliable  incentive in a certain sub-population of kids  

26:51who have certain abilities and so we may that may  be one of the things we decide um during all this  

26:59uh study period um okay should we go to the next  and thanks and do put your questions in the chatDevelopment

27:08yeah okay so the next um the next section of  the presentation i’m going to talk a little bit  

27:14about the development so where where did we come  from and then talk a little bit more about the   fda-funded study so the story of the orca really  starts with the foundation for angelman syndrome  

27:25therapeutics are fast so they came to the center  for health measurement they were looking ahead at   the pipeline seeing all these exciting potential  therapeutics coming through and they realize that  

27:35they needed a way to measure communication ability  and so the the context of use is what it’s called  

27:42but it’s basically what do you want to measure  and how you want to measure it was really specific   so they were looking at individuals with angelman  syndrome from two years old through adulthood  

27:51they wanted a broad range of communication  behavior so across expressive receptive and   social communication we knew we wanted it to be  the any measure to be for a clinical trial so  

28:02we wanted a set of items that were going to  give us a quantitative or a numerical score  

28:07we wanted to be able to identify changes  pre or post treatment we also were expecting  

28:15changes so the the therapeutics that or the  medications that this was really targeted  

28:20for this measure was for therapeutics that  were going to change or fix the genetic  

28:27issue in the brain so there would be capacity  building and then subsequent skill building  

28:32so we didn’t need communication ability  to happen changes in that to happen right   away but within a larger follow-up  window is what we were thinking of  

28:42now there’s a lot of text on this slide but i i  just wanted to point out two major limitations  

28:47so we went to the literature we looked at what  was out there to measure communication ability   for kids with angelman syndrome and there  were a number of limitations that really  

28:55justified use of a new measure the first was  that parents and caregivers were not involved  

29:01in the questionnaire design so that qualitative  the interview portion of the support was missing  

29:07and for a lot of the measures the um individuals  with angelman syndrome would score at the floor  

29:14especially those that were developed as  screening tools for typically developing children   they would squirt the floor which means you  couldn’t really tell differences if it changed  

29:24right there was not a lot of radiance there was  not a lot of ways to see those subtle movements  

29:30so what we used was the typical approach for  developing outcome assessments this follows the  

29:36fda 2009 guidance as well as other established and  published guidances out there and how to do how to  

29:44develop high quality clinical outcome assessments  so what we did is we started with talking to  

29:49people we sat down with caregivers as well as some  clinicians who worked with these patients and we  

29:56talked to them about communication what does  typical communication look like what are the   behaviors that you notice what really matters to  you we took all of this rich detailed information  

30:08and we built a conceptual framework so this is  just mapping out what communication ability looks   like to care from the perspective of caregivers  we take that framework and then we use it to  

30:19develop our survey questions this goes through  multiple rounds of review we had parents and  

30:25relatives of individuals with angelman syndrome  meeting with us regularly on the study team   to check in to make sure it was making sense when  we had the measure in a pretty good format we went  

30:36ahead and we tested it so we sat down with parents  again we gave them the measure and we said you  

30:41know how is this working for you to understand it  do you have questions is your experience captured   within it we also did a translatability review so  we gave it to a translation company and asked them  

30:53to assess how well it could be translated if we  wanted to do that in the future and then after we  

30:59had updated and modified the measure a number  of different times we gave that version to a  

31:05larger group of folks to get some quantitative  or numerical data about responses that we could  

31:11perform analysis on now i’m not going to give you  too much of the results from angelman syndrome but  

31:18i did want to show you this so this is one of our  first draft conceptual frameworks so this is based  

31:24on interviews with 22 caregivers six speech and  language pathologists and the major takeaway that  

31:31um that we got from this was that communication is  really complex right there was a lot going on here  

31:36that was being missed by existing measures there  are a lot of things that um almost every child did  

31:42in some way like requesting or seeking attention  but there were other things that were less  

31:49frequent but still really meaningful right so  telling stories asking questions or responding  

31:55to complex directions we also like we had  talked about before we heard about modality  

32:01so how somebody communicated really mattered and  we heard things like sounds signs and gestures  

32:08using assistive technology whether high-tech or  low-tech as well as eye gaze and use of the body  

32:13all of those things really mattered so we needed  to integrate those into how we asked about   communication and not just rely on verbal speech  so at the end of that formative work we ended  

32:24up with the first version of the orca measure  assessing a broad range of communication concepts  

32:31we really calibrated it to the ability  levels that we saw so the content matched  

32:38all of all the data that we gathered supported  that the content so what we were asking about   really matched what parents experiences were  communicating with their child we set it up so  

32:48that the parent or the caregiver could complete  it independently so we were using their direct   responses without anybody else um you know helping  out and it was relatively quick to complete  

33:02now when we were at that stage of the of  the project we we had a lot of interest in  

33:09the orca measure from within the angelman  syndrome community but also outside of the   angelman syndrome community so a lot of these  limitations that existed in existing measures  

33:20were also affecting the measurement  of communication ability for other   neurodevelopmental disorders and a lot of the  groups that were contacting us were really also  

33:29focused on this clinical trial space so how do we  measure communication within these clinical trials   so we started thinking about how do we scale up  how do we create efficiencies across different  

33:41diagnoses in different populations so that’s  when we applied to this fda funding program  Project Overview

33:49which was focused specifically on coa or  clinical outcome assessment development  

33:54we were so lucky to partner with combined brain  in this project so the ultimate objective of this  

34:02study is to really promote patient-centered  clinical trials by focusing on the measure of  

34:07communication ability using the original version  of the orca measure as a guide so this is a phased  

34:15award we have a two-year planning phase and  a three year implementation phase so the full  

34:21project will be five years the first aim is a  qualitative aim so we’re looking specifically  

34:27to identify the core concepts and levels of  communication across the 12 neurodevelopmental  

34:33disorders that are part of this project so we’re  interviewing caregivers basically for a1 lots of  

34:38caregivers we’re talking to folks about how they  communicate with their child what do they see  

34:44what’s important to them the second aim that’s  listed here is basically just us synthesizing  

34:50the information deciding what we need to do  in that next three year phase of the project  

34:56this figure just shows um the data sources so  on the left in blue we’re doing qualitative  

35:02interviews with caregivers we’re also  collecting any data that’s out there on  

35:07um communication measures and then we’re  using that to identify where the gaps are  

35:14in the measure where do we need to add  questions where do we need to modify

35:20this figure shows the organizational chart and so  this is shows you all the different stakeholders  

35:26that are involved so i would say patient advocates  parents caregivers are our primary stakeholder for  

35:31this project right we want to measure that  really speaks to the family’s experience and   is meaningful for them we also have a number of  clinical experts obviously the health measurement  

35:43team we also have an external technical advisory  committee that has folks from industry the fda  

35:50patient advocates all involved so this is  a heavy communication style project where  

35:56we’re getting feedback on every level of the  project including how we’re doing the interviews  

36:04what does the data mean and what do we  need to do next so i’ll stop there and  Qualitative Interviews

36:10turn it over to leslie to talk a little bit  more about aim one and what do what do the  

36:16qualitative interviews look like and what can the  parents expect from the qualitative interviews

36:25thanks okay um you can go home chrissy

36:36okay before um the caregivers ever get to the  actual interview process they first have to  

36:42go to through the screening process um and  that’s essentially to determine if they’re  

36:48eligible or not to be a part of the study the  entire screening process if they’re eligible  

36:54is around 20 minutes so first we just schedule a  screen call with the caregiver and we complete the  

37:00screening to determine their eligibility  the eligibility criteria is pre-approved  

37:07by the research team and also the stakeholder  group so they get together and they determine  

37:12the eligibility criteria together in order to  complete the screening we remind the caregivers  

37:20to have a copy of their child’s genetic report and  have reliable cell signal and internet connection  

37:28um during that process while asking  those questions if for whatever reason   they’re deemed ineligible we thank them for  their time and and stop the screening there  

37:39if they are eligible we get their verbal  consent and demographic information  

37:45get things like their address to send them the  orca survey and schedule the interview go ahead

37:55now the actual interview is around 60 minutes  and it’s in mostly comprised of two parts  

38:01the first part is the concept elicitation portion  which i will explain more of what that is in  

38:06the next slide and then the second part is the  actual completion of the orca survey that they  

38:12should have received in the mail beforehand and  questions follow-up questions about the survey

38:21so the concept solicitation portion   concept elicitation is essentially a conversation  where the interviewer asks open-ended questions  

38:31to understand what concepts are important to  them like christy has been mentioning and um  

38:38uh so during this this part of the interview  we’re asking them questions like what typical  

38:43communication looks like for them and their child  what are their child’s communication strengths  

38:50open and question open-ended questions like  that this is an example of an interview that  

38:56was conducted earlier for this project and the  caregiver was talking about how their child’s  

39:02communication how they communicate their needs  and especially it tends to happen around food  

39:10and listening to the caregiver talk about what  typical communication looks like for them the  

39:16interviewer also picked up on the modalities of  communication that their child uses and in this  

39:21case it’s gestures the interviewer also talks  about other possible modalities of communication  

39:28that the child may use like sounds words or word  approximations um according to the caregiver’s  

39:34response we then learn that the child uses some  sounds and word approximations to communicate  

39:42um this is just one example like i said we also  ask a variety of other follow-up questions to  

39:49learn more about their communication with their  child and also what is meaningful for them

40:00go ahead and then after this we transition over  to actually complete so we give the interview the  

40:08caregiver some time to complete the survey after  they’re done we ask them some follow-up questions  

40:14about their overall experience so this includes  things like what was irrelevant what was missing  

40:23what was confusing or hard to understand  and also specific probes about specific  

40:30items on the orca survey that we’re wondering  whether or not they’re relevant for that group  

40:37um so and this is another example where we ask  those follow-up questions and the interviewer  

40:44is asking if they consider one of their one of  the modalities included in the survey as their  

40:50child’s primary mode of communication and the  answer that yes they do it’s gestures and signs  

40:56and then the interviewer follows up and asks why  and then they we learn that they consider it their  

41:01primary mode of communication because it’s the  one that they use most frequently um we also ask  

41:08caregivers about like i said anything that they  found difficult um hard to understand about the  

41:13survey um we asked them about the recall period so  is the 30 day time period appropriate for them um  

41:22and like i mentioned at the beginning for the  most part this whole process takes around an   hour to complete however caregivers love to  talk about their kids so sometimes it may go  

41:32over but we make sure to check in at the one  hour point and be respectful of their time

41:38all right yeah thanks so much leslie all right so i just  have two slides left and then we can pause for  

41:48questions and discussion that sounds good okay so  where are we currently within the fda funded study  

41:56so we are within the two-year planning phase  we have a lot of qualitative interviews to do  

42:01with caregivers the way that we’ve set up that  two-year planning phase is we’re rolling out our  

42:07interviews in cohorts so each cohort consists  of three different neurodevelopmental disorders  

42:13we have a study startup period with each patient  advocacy organization where we work with them on  

42:20how should we recruit caregivers what questions  should be asked who is eligible who should not be  

42:25eligible and so right now we have finished cohort  one they are in data analysis and we should be um  

42:34reporting and talking about some data by the end  of the summer so we’re really excited about that  

42:39we are currently enrolling in cohort two i would  say we’re probably halfway maybe a little more  

42:45than halfway for cohort two and cohort three  is on deck so we’re working with those groups  

42:51we’re gonna start working in june more intensely  about their specific protocols as well as  

42:59our qualitative team will listen to a  presentation from them to talk about  

43:04their experience caring for children with  each of these neurodevelopmental disorders  

43:09so what is the end goal of this  project what we’re hoping at the end   that we will have a family-centered measure of  communication ability with documented support for  

43:21use across multiple neurodevelopmental disorders  we’re hoping that this measure is high quality  

43:29will be accessible meaning it’s out there for  people to use within their natural history  

43:34studies and their clinical trials and we’re hoping  that ultimately it the content will be meaningful  

43:40to multiple stakeholders um with and  centered in the family’s perspective

43:48so finally just to wrap up this is a picture  of our duke team we have you know multiple  

43:53different layers of expertise in here  qualitative quantitative communication  

43:59um all of the things and then um these are some  of our partners and our collaborators and i i  

44:05also want to take the take every occasion to  express my gratitude to all of the families and  

44:12caregivers and relatives who participate  both within the study on the study team  

44:18in the stakeholder engagement group meetings that  we have but also the families that participate in   the interviews and donate their time to this  um we’re so appreciative and we’re hoping  

44:29our entire goal is to do you right and make  sure that we have a measure that really   speaks to your experiences and meaningful to  you so i’ll go ahead and and stop sharing now um  

44:40so that i can see folks faces of people unmute  and we can have some questions so thank you so  

44:47much um it’s really great to hear the whole  presentation all at once i have a question i  

44:54want to ask and then i want to open it up to other  people so um at the end of the five-year process  

45:03will this only this survey only be considered  valid in the 12 disorders plus angelman syndrome  

45:12or will it be considered valid and usable  across a lot of neurodevelopmental disorders  

45:20yes that again is a great question so i’ll i’ll  speak i have two points that i want to talk about  

45:27so traditionally in measure development it’s  been done in silos right where we you know  Measure Development

45:34we focus on angelman syndrome we gather all this  evidence we have a measure for angelman syndrome   then we maybe move to the next one and then the  next one right one of the one of the benefits to  

45:45doing it all at once is that one of our goals is  to create efficiencies in this process right so  

45:51what is the information that we need to know about  a rare disease to make a determination if the orca  

45:58is going to be useful or not right what are those  questions that we need to ask parents and families  

46:05so the hope is that we’ll create efficiencies  and we’ll have a ton of data from the 12  

46:11different disorders who are part of this and we’re  hoping that all of that will make it easier for  

46:17other folks to determine if the orca is going  to be meaningful and valid and reliable for them  

46:23so i think ultimately our goal is broader  than the 12 groups now i think one of the  

46:28key pieces right is if a measure is going to  be used within a clinical trial you need buy-in  

46:33from all the stakeholders including the fda or  the regulatory folks that are involved right  

46:39and i think that’s one of the things that made  this specific grant um opportunity really exciting  

46:45for our team right because i can’t obviously  i cannot speak for the fda but on our side  

46:51the grant is set up where we have you know  this interaction with the fda at multiple  

46:56parts of the project on a regular basis so that  we are more likely at the end to have buy-in from  

47:03the fda right because we’re we’re allowing  for we’re able to integrate their feedback  

47:08throughout the process and so that was really  what attracted us to this this opportunity

47:16and what kind of feedback do you  think the fda has had i mean what are  

47:22the concerns that the fda has about making a  survey like this what what does the fda care about

47:32ooh that’s a good question now i’ll say again that  i cannot speak for the fda and their priorities  

47:38and um but i will say that one of the  things that we’ve talked a lot about  

47:46is the goals for the qualitative work and what  what type of questions do we need to be asking  

47:54parents to make sure that the content is  matching their experiences we’re able to  

48:00pick up on differences between neurodevelopmental  disorders and how individuals are communicating  

48:06and we’re thinking about meaningful change  right so what does that change look like  

48:11whether an individual is getting better which  we’re hoping within the clinical trial but  

48:16also for diseases that are more progressive does  is meaningful changing mean that they are stopping  

48:23loss of skills right there’s they’re  keeping all the skills that they have now   and so i think on our end my impression  is the content is high priority for us  

48:34as well as a lot of the discussions we’ve  been having as well as that meaningful   change so what would change look like how do  we conceptualize that um that’s my impression

48:46um i do want you all to put questions in the  chat but if you don’t i’m going to keep asking  

48:53um leslie what i want to know is you’ve  been interviewing the different disorders  

48:58how different are they between each  other are you having a lot of surprisesInterviews

49:08um i think it’s a little too early to tell  but so far in just my individual interviews  

49:15um because there’s i know nicole’s on the call and  she’s done interviews too and we have another um  

49:22colleague who also does interviews they have  been very different they have been like from  

49:28personally from caregivers that have children  that are very non-verbal to children that speak  

49:35multiple languages and full sentences  so it i think it varies a lot  

49:42um for my personal experience from group to group

49:48um so there’s another question jayada asked the  good questions and her question is why does the  

49:55fda seem to value differentiation across a  disorder which may or may not be significant

50:04so again i i cannot speak for the fda um but i  what i can say is i think one of the questions  

50:10that we hope to answer with the qualitative  work and why we’re doing you know eight to   twelve interviews within each neurodevelopmental  disorder is so that we can speak to heterogeneity  

50:21or differences both within a group across  caregivers but also between different groups  

50:28right are there differences or do the are there  core set of communication components that are  

50:35relevant to everyone and i think one of the things  that we’re finding that leslie just spoke to is  

50:41that there’s a lot of variability in communi  and how individuals are communicating and what  

50:48level they’re communicating at but we are seeing  a core set of components that are coming up across  

50:54groups right so the way that the the orca measure  is currently set up we have things like refusal  

51:01right say no to something but there’s a lot of  different ways somebody can say no they can they  

51:06might cry or fuss if they don’t want something  they might push it away they might look away  

51:12they might throw it across the room right or they  might say no or they might shake their head no  

51:17or they might have a symbol or a gesture that  says no so that heterogeneity and how is the  

51:23person communicating that feels like really  important um to have on the measure also within  

51:31these larger concepts that are resonating across  different families right and so it does at this  

51:38point again it’s still really early because we  have not finished analysis for our first cohort   which has three neurodevelopmental disorders  and we’re currently in interviews for the second  

51:47cohort but it does feel like there are core  sets of components that we should be measuring  

51:54and our and our job is to really get at  the different examples of how individuals  

51:59are communicating for that purpose and making  sure that matches with the content on the orca  

52:05so i would suspect this is just my impression  again that moving forward if we have similar  

52:14clinical presentations the underlying genetic  mechanism is not what’s really gonna matter to  

52:21the measurement itself and does it match with your  your parents and your families it’s really about  

52:28um you know how somebody is communicating and  what that it’s it’s more downstream i think than  

52:34you know the more physiologic changes that we  might see with a genetic treatment communications  

52:40more aspect of function so we only have one time  for one more question so it’s going to be mine  

52:48and that is the question i get a lot about this  project which is it’s just a survey why don’t  

52:55you just send out this survey and see what people  say and then you analyze it and then you’re done  

53:03why aren’t we just sending out the orca that you  already wrote for angelman syndrome or developed  

53:10and already tested in angelman syndrome why  aren’t we just using that why why do we have to go  

53:15through this big long involved process yeah well  i think the hope is not everyone will have to go  Whats Next

53:22through this big involved process but i what we’re  trying to answer right now is all related to that  

53:28validity component right and the only way to know  if the content of the measure so the questions  

53:36that are on it are really getting at what we want  to measure and really resonating for families  

53:41is talking to people that’s the only way we  can do that and right now in a lot of a lot  

53:47of the neurodevelopmental disorders what typical  communication looks like from the perspective of  

53:52the caregiver is not established so i think as  parents you know and you see other families and  

53:58you really know what communication looks like  as researchers clinicians and clinical trials  

54:04and the fda we don’t know what’s happening in  your homes so we to know that the measure and  

54:10to really document right like we have to have it  written to share with our stakeholders to really  

54:15know that the measure is hitting the right place  in communication ability we have to hear from  

54:21parents about what communication looks like and  so that in the fca guidance that was published  

54:27in 2009 that like aspect of content validity  and other you know other published guidances  

54:36quantitative or numerical evidence can  support validity but to really actually  

54:43um evaluate content validity you need that  qualitative work so you need the interviews with  

54:48caregivers so you have to actually ask people what  they think you have to yeah we have to ask people  

54:55what is typical communication like for your child  and what their life is like what their kid is  

55:02okay so here’s the very last so we still have  four seconds left so the very last thing is  

55:09could you you know we’re always talking about  measures that are our kids are below the floor  

55:16um will you also is it will it be possible  to graduate from the orca so for example  

55:23if you’re if you’re a child who gets better and  is able to learn to read and write and spell then  

55:33will the orca no longer be the right measure for  you will that mean will you now need to use a more  

55:41standard school-based measure after that yeah i i  think i think so so this is one of the questions  

55:49that we’re we’re hoping to start to answer in  the planning phase based off the information  

55:55that we’re getting from interviews but is also  something that we’re going to have to tackle   in the second phase of this project is what is  that upper bound right when do people graduate  

56:05and how can we link scores so that we know what  it means and we we can track people as they  

56:12increase their communication ability and there are  existing measures right that do a really good job  

56:17of evaluating typical communication development  in a way that’s we have a lot of normative data  

56:24a lot of reference data we know how to interpret  it right there’s measures that do that really well   so i think one of the goals that  i’m hoping to do with with the  

56:32more numeric data that we’ll get in the next phase  is really see how we can do that how can we link  

56:38measures and how can we keep track of folks if  they’re jumping from this one to that one yeah  

56:45well i think we’re out of time but i would just  really like everyone to turn their cameras on  

56:51for a second just so we can see everyone and  if you can and i’d really like to just thank  

56:58all the combined brain groups who have been  a part of this and i would like to thank all  

57:03the people who work for combined brain who  have been a part of this and then above all  

57:08i would really like to thank the duke team  and everything that you’re doing thank you