DONNA STROBINO: Okay so we're going to have group one report; they've done both of them. So Cathy's going to come up and tell us what she thought.
CATHY: Okay. The first one was the percentage implementation of activities from the State Pediatric Asthma Plan. And we basically in question number one, it was bad on all counts. There were some really bad grammar at various points, if nothing else, and it wasn't clear. It was open to different interpretations because we couldn't figure out is there one plan, are there two plans, what are the plans, what are the components of the plans. So it may include unrelated items but we don't know because we don't enough about both of the plans. Depending on if you got more information it might be a group of related items and it might have a unique process that you were describing but we have no way to tell that. Number two, again about numerators and denominators, under the definition there seemed to be two plans.
Then when you get the numerator and denominator, it talks about a Pediatric Strategic Plan, again without any definition of what's in the plan or how many activities are in it. This could certainly be restructured by providing additional information, attaching notes, attaching a list of what's in the plan, or the two plans, and if there's two, how would you merge them together? In terms of number three, whether appropriate data sources identified but if not, no. Again, they mention something but it's not defined enough that you don't know what is in the plan it certainly could be remedied. And then what could be done to improve the performance measure, how could it be revised?
How could the numerator and denominator be revised? There were two thoughts; one, when we had looked at the goal they seemed to be particularly concerned about, hospitalizations, one, just as a radical thought, if you have hospital discharge data in this state, measure the hospitalizations directly. You know, skip the plan part, and look towards an outcome. Assume, however, if they wanted a capacity, that they really needed a plan for this, you probably rather than looking at a raw number of percentage of specified activities, you could do that but you would need to attach them and you might want to weight them if the plan had something from convene a group of experts to if another part of the plan was to train every pediatric emergency room on triage, that that would be much more complicated and more difficult so that you might just want to not just give each of the activities the same percentage weight. You might want to develop a scoring scheme and if you did that you would want to describe it and have an attachment so that people could see how you had come to the score. The second one which we--we did there too.
DONNA STROBINO: Oh wait, I'd like to--Is there any other--Are there any points of discussion here about that first one that anybody else wants to offer? Okay I think that one of the points that Cathy made which is a really good point with regard to this example is that one of the problems with it is there isn't enough information that's given about how they're going to actually develop the measure and the components of the plan in order for you to evaluate whether or not it is a good measure. I think that's a really good point to see is that that's important. It's important that those details be included. Did group one have any other comments? All right Cathy, great. That was great, by the way.
CATHY: Okay. The second one was the percentage of high school or you--we think--we bet youth--who feel supported at school. So again, typos are critical in proofreading these. This one, one as we read through this it was clear that the concept here, we think people were, sort of, looking at the search institute and a methodology that there were sort of a lot of rationale but if you didn't know that, which one member of our group, a couple of members did and the rest of us didn't.
It was not at all clear how you got from the goal of reducing risk behaviors, particularly abstinence from sex, alcohol, and drugs, to feeling supported at school. So the link between the measure and what you were trying to get at was not clear from what was presented. And it was obviously open to different interpretations; what does it mean to be supported at school? In terms of the numerator and denominator and the data source issue, they reference a survey but we don't know if this is a one-point survey that they developed. There are questions like this on YRBS, which would at least be tested and you would know what you were talking about. There might be some other national survey that they were using. But you don't know that. You don't know, therefore, whether, as a number of people said, high school students will tell you anything, how do you know that a survey answer has anything to do with how healthy they are. And that it wasn't clear whether all the students in the state were being surveyed, if this was a representative survey. What you just said, our basic thing here, you can't give too much information.
If this is a survey that's been well tested and you're doing it in what has been designed even as a representative sample, then that's good. You would have statewide that people could decide how they regarded that. But that you also, if it is a weighted survey, you would not be going with a percent, you would go with a weighted score. The unit is the final percentage based on the analysts that give you the data, so that was just a fine point. But it could be important but it could also be, like, completely out to lunch.
UNIDENTIFIED SPEAKER: (inaudible). It's a question on the Communities that Care survey.
CATHY: We have no idea. If they say their data source is Adolescent Health Program, period. And they mention Search Institute in the rationale about why this is important. So I don't know.
DONNA STROBINO: Are there comments about this? You do know that if you go to tab eight you'll have that information in front of you? Okay. Because I wasn't sure if every--everybody had gone there. This one's sort of an interesting example from my perspective and that is how is the public health system going to affect whether or not children are supported in school. I mean, to me, I understand their ultimate goal and it makes a lot of sense. But is what they've written actually an intervention that they have control over? And I'm not convinced unless they have some important programs they're implementing in the schools that you actually can have some control over this particular measure. So from even from that, from the face of it, it's not clear to me that it's an appropriate measure.
UNIDENTIFIED SPEAKER: I think that this is where it--what the discussion had gotten to our group about the content that you need to interpret these. And in Vermont we have a question that it is set though, the YRBS and the Department of (inaudible) has a lot of mentoring programs, etcetera, etcetera, so there is a system to be put in place. That's why you need the people who knew the content sitting around the table as you're discussing these.
DONNA STROBINO: Good point. Okay, good. So that there actually may be a system in Vermont, they actually have a system where they might be able to affect this so it's important to have those details. Group two, I think you got through the first one? Is that right? Yeah Maureen, very good. Did you go on to the second one?
MAUREEN: Well, we didn’t (inaudible).
DONNA STROBINO: Okay. Well if you want, if you think there are any comments you want to make on the second one, that'll be fine too, but just start with the first one.
MAUREEN: The performance measure was the degree of improvement in the bureau's programs participation and prevention initiatives that impact conditions and complications affecting children with special healthcare needs. And I guess our first--We didn't--We were concerned about what participation meant, what the extent of prevention activities would be, are we talking primary, secondary, tertiary prevention. And then what the definition of children with special healthcare needs might have been within this particular state. And the goal was to increase by five percent the number of bureau programs containing prevention components. So we felt that the measure wasn't clearly stated and it wasn't concrete. And certainly, in our group it was open to different interpretations and potentially combining several related and unrelated items and it wasn't very focused for the reasons that I indicated. When we got to the numerator, they described the numerator as the number of appropriate bureau programs, not including WIC, which include a prevention component.
Well, no. The appropriate bureau component might be every bureau, probably, in the Bureau of Family Health but that was--that was somewhat unclear. And at least in some of our states, this wouldn't be something you could measure over time because the bureau, the number of programs in the bureau would be changing by the week. So I'm not sure about the numerator and the denominator. I think the definition of the sources, we don’t know, even if we've defined and presumably they have defined these things in some--someplace else. But if we were able to define these, we don't say where we're going to get the information. The data is going to come from the Bureau of family Health but how would we define it? The number of people who give grants, the number of people who have contracts, the--are we going to interview the program manager's health, are we going to look at the website, how are we going to come up with the prevention programs as they were defined.
We thought, although this wasn't the performance measure, we thought there might have been other ways to approach this particular activity that they're trying to get at, we could improve it by describing these, defining these things better; children with healthcare needs, defining participation, defining prevention and particularly trying to focus those issues. And then looking at something more specific about what participation meant. For instance, giving grants, seeking funding, applying for grants, and so we weren't--we didn't feel like we really got at the intention that they were trying to look at in their discussion of significance of the problem. Does anybody else in our group have a comment? Did we--No?
DONNA STROBINO: Are there comments from other groups?
UNIDENTIFIED SPEAKER: I guess just looking at the measure and the goal, there is, like, I interpreted that the degree of improvement as an issue of quality versus where the goal is, the degree and number of programs. So it's like (inaudible)we’re just talking about numbers.
DONNA STROBINO: Okay, so there's a disconnect between the goal and the measuring. One is talking about quality and one is talking about numbers. Yeah?
UNIDENTIFIED SPEAKER: And that's my profession (inaudible).
DONNA STROBINO: Good. Okay, good, yeah.
MAUREEN: But I think that's important because a five percent increase in the number of programs in the bureau would probably be one. This would probably be achievable.
DONNA STROBINO: Well so that's probably not then a responsive performance measure where it would be responsible, we'd actually be able to see something that happened, good.
UNIDENTIFIED SPEAKER: What also strikes me as curious that (inaudible) being very clearly excluded from this and with their significance talking about folic acid and (inaudible) prevention, it screamed to me WIC needs to be there. But, you know, so there needs to be, I guess, (inaudible).
MAUREEN: It was probably made to define, I mean, to explain why they felt they needed to exclude WIC in this particular one.
DONNA STROBINO: Good Point.
UNIDENTIFIED SPEAKER: It also, as you've read, until you got down to the end again, it sort of lost it's focus around children with special health needs in the middle. And so if you were looking at prevention that was particularly important to children with special health needs, which is perfectly legitimate, you should have kept saying those same words in the definitions. So that if you were going to exclude programs as inappropriate because they had nothing to do with children with special health needs, then that would be fine. But it's like the shorthand, then, get you confused as to what you're counting.
DONNA STROBINO: Um-hmm, good. And you could make that longhand either by using an asterisk and putting it on the bottom, or including it within the numerator and denominator in the way you set it up.
UNIDENTIFIED SPEAKER: That's (inaudible).
DONNA STROBINO: Right! So there's a lot of different ways you can do that but the point is it needs to be there. Right, that precision needs to be there. Good. Maureen, you don't want to say anything about the next one?
MAUREEN: I can tell you what it's--we can talk about it.
DONNA STROBINO: Do you want to tell me where you started anyway?
MAUREEN: Yeah.
DONNA STROBINO: Yeah.
MAUREEN: This one is the degree to which statewide data are available to monitor health related behaviors among youth. And the goal is to monitor priority health risk behaviors among youth and they have--they identify in their definition eight of risk behaviors and then they propose to assign a score of one or two, I think based on how available the data is, whether it is just available or it's been available for a while. Our discussion, well I think we felt it wasn't really clearly stated in concrete terms but they then did define in their definitions it became a little clearer. There really isn't a numerator and denominator but it is a score that presumably would be turned into a numerator and denominator over time. We don't know where the data is going to come from. These sound like the kinds of things we would get from YRBS but it doesn't say that. And presumably in this department of health education, mental health, mental retardation, and substance abuse services. There could be different sources for alcohol abuse in young people. So, again, it may have been very clear to them but it wasn't clear to us, and so perhaps having a better or clearer definition and better description would have made it more appropriate.
DONNA STROBINO: That's actually a good segue into the next group because you did a data one next, two, right? Yeah, so group three. Did you do two? Or did you do both of them or just one?
UNIDENTIFIED SPEAKER: We did one. The second one I didn't have the data to analyze that.
DONNA STROBINO: Okay, so we're going to just do five, right?
UNIDENTIFIED SPEAKER: Five, yeah. On that group five, the first question is what is supposed to be percent of low-income children ages one to 11 that consume nutritionally adequate diets. The goal is to increase their proportion of children at risk who consume diets that meet the recommended dietary guidelines. Under that definition what we found is that the performance measured, the measure is clearly stated because they actually define what they mean by low-income children and they give a guideline. They defined what they meant by children and they identify the age cohort of children and then they also define what they meant by nutritionally adequate diet, which is the conception of the recommended number of servings of all food groups in the food guide pyramid.
So when it comes to question is it clearly stated? We said yes, it is. Is it open to different interpretation? We said no, based on this statement that they have. Does it combine several related or unrelated items? We said yes because if you look at what they were defining on that data source, they're going to use the WIC data and they're also going to use data from grade four children at school. So it did combine because of the different factors that can easily militate against that. And also when you look at the data source they were saying this is going to be reported by women, mothers of the children, at WIC data and that is a different program than they (inaudible) for and the (inaudible) becomes putting the data by children so, you know—
DONNA STROBINO: Right.
UNIDENTIFIED SPEAKER: You cannot really do that. And the outcome, do they have a focus on a unique outcome? We said yes the focus is on the nutritional diet so there is a clear focus but it did not tell us to what extent will the percentage increase be. So we don't know the degree of increase enough for you to measure, there is no baseline; as a result it is difficult to really know what is your baseline and what is the change that will be made, you know enough for you to measure the degree whether they did it or objective? We said no not. On that second issue the numerator and denominator clearly specified and directly related to the measure because they need to change the language. Why we said that is because the data, as you see it here did not clearly measure the goal as stated. The goal is saying to increase the proportion of children at risk who consume diet that meet the recommended dietary guideline. But the data they're using are data from WIC reported by the mother and then children only at the fourth-grade level, so that doesn't give you the whole universal of children that they're trying to measure.
So the issue is could you generalize based on just the fourth grade response and then you look at the fourth-grade people? There is a selection bias because it's not even the whole fourth grade; it's only fourth graders who for lunch, really, agreed to participate in a nutritional program. So you see that you cannot adequately measure that. Number three, are appropriate data sources identified and the question, if not--We need to pick one data source because they are using two data sources so it's either they break it down to zero to five using WIC data, which is good in a way because we target low income children zero to five. So it gives you a good universal database to come up with your baseline. Or they can use the fourth grader and look at that fourth grader, kind of participation but if possible, we said explore including other fourth graders so that you can then measure the impact, what are the impacts of those people that are really participating in this program versus those that are not. So you've still got your denominator will be the global universal database.
And at the end of the program it said that the size, because of this that I discuss it, size is not adequate for them to actually effectively respond to the goal in question as stated. And there is no periodicity; we don't know when the data, whether there is any defined periodicity of when this data will be collected so that you can measure this timeline versus this timeline to see whether there is any confounder. That was not explained. The fourth question, what could be done without--Yes, something could be done. If you apply all of these critiques, you could actually enhance the measure, change the measure, or change the goal. You have to do something with it too. You’re measuring the goal or you're changing--you're coming up with a different measure to do that. And you have to break the groups so that you're not combining apples and oranges. And then we said if you're using grade four, again include the whole (inaudible) that I have here, a good universal as your denominator in order for you to tease out any impact of selection bias based on the participation. And I think that's really what we said.
DONNA STROBINO: Good.
UNIDENTIFIED SPEAKER: Any comments from my group? Yes Jane?
UNIDENTIFIED SPEAKER: You did a great job!
DONNA STROBINO: That was a tough one because on the face of it, it actually--When you first look at it you think, gee this is an interesting measure. The other issue here is cost. The food frequency questionnaire takes about 15 to 20 minutes to be implemented. And that's a long time and that's going to be expensive just for one--one particular outcome. And the other issue here is I'm not sure what an adequate diet is. To me I wasn't sure exactly how they were going to achieve that, although I do know that there are different--there is different information you can get about nutrients and other measures that would bring you back to that adequate diet.
UNIDENTIFIED SPEAKER: They said that we used the Food Pyramid, you know, like the WIC has the Pyramid?
DONNA STROBINO: Right.
UNIDENTIFIED SPEAKER: So we said (inaudiable) that you find what the source that needs to capture that and then you just accept it, whether it is adequate or not.
DONNA STROBINO: Right.
UNIDENTIFIED SPEAKER: Because we have a definition of—
DONNA STROBINO: Yeah but I'm not sure if that's necessarily the way that questionnaire is usually used. Because it's actually used to estimate nutrients that are actually consumed by the individual who reports it.
UNIDENTIFIED SPEAKER: So you wouldn't know that it would not affect the baseline.
DONNA STROBINO: Well no, I understand that but I mean this is again from my standpoint, a situation where once you've gotten to the point where you say okay, let's see what we're doing. Then we still have to go back and say, okay, how long does it take and what are we actually going to get? Okay, yes?
UNIDENTIFIED SPEAKER: The other thing that was interesting was that this was clearly an iterative process.
DONNA STROBINO: Right.
UNIDENTIFIED SPEAKER: Because we felt that the steps were good but when you tried to put them together they didn't fit.
DONNA STROBINO: Right.
UNIDENTIFIED SPEAKER: So then you had to go back and either rewrite the performance measure or use that--use the performance measure but find different data sources or whatever.
DONNA STROBINO: Good. So you had to be consistent.
UNIDENTIFIED SPEAKER: Yeah.
DONNA STROBINO: It’s sort of, like, yeah so if you set out a goal, you need to be consistent all the way through on the rest, good. Group four, you did both of them? No, you did one, so it was group five who did both, right? As well, I think? Yeah.
UNIDENTIFIED SPEAKER: So we're going to look at State Performance Measure Seven. And the measure says percent of female clients suspected of being victims of relationship abuse. We didn't get beyond this measure because it really needed a lot of work in our opinion. So first take a look at the measure and then look at the goal, the goal says increased detection of and appropriate responses, okay that's two things, to women and adolescents, is that female adolescents? Seeking health services who are suspected victims of relationship abuse. So, a couple of things, we didn't like the word suspected, for one, we thought maybe screened as at risk or something like that would be a better word, determined to be at risk.
It wasn't clear to us what they were looking for here. The measure that they were looking for, sure we can provide the numerator and denominator, the number of female clients suspected of being victims of relationship abuse. We can figure out in our own state how we want to define that, should we use a specific tool and instrument to do that, perhaps. Number of female clients receiving services from Title V, X, and XX clinics as the denominator, okay we can do that but that doesn't really get at what the goal is, increasing the detection of and appropriate responses. So we thought that the goal should be the anchor, first of all, not the measure.
DONNA STROBINO: Good.
UNIDENTIFIED SPEAKER: So anchoring from the goal we need to first narrow that goal down to one thing, not two, and be a lot more specific in that goal, even though it's a goal, not an objective, it should be one thing. So do we want to increase the detection of women and female adolescents seeking health services from Title V, X, and Title V, X, and XX clinics who are determined to be at risk for relationship abuse? Or do we want to increase the appropriate responses to those? Or do we want to know the numbers who were screened? Or do we want to know of those who were screened, how many are at risk? So there are all these questions and it just was not clear to us what they were trying to do with this performance measure. And so we went round and round on all of this and really picked it apart and you could write, probably, at least three good, solid performance measures from just this one performance measure. So rather than, you know, throwing out three different ones, I think that probably shows where our discussion really went to. Do you have anything to add Sally? Yeah.
SALLY: Yeah. Just basically that that then we also--It was one of these, again, iterative process that we keep talking about, about what do the program people really want in their program? What are they trying to do? Because, you know, do you want to increase screening? Do you want to increase screening referral? Do you want to do men too? Like in the Title X serves men. So yeah, we could get three good—
UNIDENTIFIED SPEAKER: Yeah. You need client visits?
SALLY: Oh right, right. Yeah, yeah we were just in a discussion of should you ask every time they come into a Title X clinic about risk factors for abuse and if the person says no the first time and comes in a week later, you ask again, they say no, as week later they say no, and the fourth time they finally maybe have developed the trust to say, well, yeah there is this little incident that we go through, blah, blah, blah. So that's what we realized, we had to go back to the program people and I know having been both, these type of questions help you figure out well, what do we want in this program and what are we trying to do. So it's a good discussion and it reminds me of what we were doing this morning of what is cultural competency? Well how do we measure it? Well maybe we don’t know what it is until we try to measure it. And then once you try to measure it we go back and then we know what we're talking about. So it's not a bad thing but it's bad if you don't have the discussion to come out with a proper end, so good.
DONNA STROBINO: Good. Any other comments on that? Both group--Yeah, Cathy?
CATHY: I was going to say, because we have tried to develop a similar measure in the other concept that we had difficulty with is that whether you're looking at relationship abuse, or sexual abuse, or both, you're dealing with an underreported reportedly a vastly underreported phenomenon and so if you look at either suspected, you know identified cases, or screened in, or whatever, you're never sure whether you're the in, you know, in fact we were, it that the more we did, the higher it would look like the abuse was going on and we didn't particularly want to look like that on, you know, the surface. But you sort of want to get at that because you think there's a lot of unreported. So we, in fact, went to something set, you know, the denominator and the numerator are so unstable because we really don't know what the true prevalence is. We're going to skip that right now and went towards the training people on the use and the documented implementation of the screening tools. In other words, are you asking somebody (inaudible)? Because that, we felt, there was no downside to that and you weren’t going to get the measure going a way that you're going to have to explain.
DONNA STROBINO: Good. That's a great example of that you developed and measured where you were and one that would be responsive to what you were doing. Because I'm not sure with this measure I can understand what I'm actually getting and what this measure is supposed to be responsive to, I think that's an issue. Three and four both have indicated iterative processes. Very important message; this is an iterative process. And so when, you know, we're thinking about in our state, what measure do we want to develop? It needs to be an iterative process of going through the thinking. I'm not saying not just thinking about where you are but actually going through the thinking at each stage of what's the goal, what is it I want to get at, am I really measuring it, do I really have the information. So, it's great. Great discussion. Group five? Now you got through both of yours, right?
UNIDENTIFIED SPEAKER: Most of the way through the second.
DONNA STROBINO: Most of the second one? Okay.
UNIDENTIFIED SPEAKER: This is a learned art, this was an excellent exercise from our perspective and that it is very much a learned art that requires, I mean, when you get such a tunnel vision as you're working on these and you know exactly what you mean and I'm sure whoever wrote--whatever state wrote performance measure number nine, understands it implicitly. We don't, lots of questions. So one of the things I've learned through this exercise is that whenever we think we've got our final new performance measure, I'm going to call up people in different parts of the country and say, read this and tell me does it makes sense! Our performance measure was percent MC Plus managed care organizations utilizing MCH data. Question number one; the multitude of questions, the first was is it in concrete measurable terms? No. Open to different interpretations? Absolutely. Combined several related or unrelated items? Absolutely.
And no it doesn't focus on a unique outcome or process. We identified that we had several definitions for MC Plus in our group. For me, that was Medicaid. For somebody else it was Medicaid clients, for somebody else it was I don't know. Then also MCOs, is that managed care for Medicaid insured people? In my state we don't tend to have that so MCOs are--that's my way of referring to the private sector insurance, versus the public sector. So we need more definition in regards to what's MC and is MCO meaning Medicaid Managed Care or private and Medicaid managed? In terms of the goal, it was not clear to us. We had difficulty seeing a connection between the request for--or the utilization of data and then the description about technical assistance, that we had a hard time making a match there. The last thing that--in this, sort of, first section was what MCH data is used and what was the purpose of this? Was this that the Title V agency felt that nobody was paying attention to the data or the knowledge that they had and they want to be utilized so they're trying to measure that contacts them. Is it that the population that these insurers have is very segmented and doesn't include the population that the agency has. What was to be gained from this? Was this to give us a role or was it to have an outcome for clients, changes in outcome?
And that wasn't clear. In terms of trying to come up with a numerator and denominator again without knowing what MC and MCOs meant and what was the desired outcome, it was a little challenging. But we came up with for a numerator the number of Medicaid and managed care organizations requesting technical assistance with MCH data from the state Title V agency. And that the denominator, we thought, would be the number of Medicaid providers and managed care organizations that the opportunity for technical assistance was marketed to. Again, we don't know what data they were referring to specifically that the Medicaid and MCOs wouldn't have access to. Another possible denominator was the managed care organization's license to serve in that state that are serving the maternal child health population.
We weren't sure if there might be some MCOs that only focus on the elder care, say, I mean, Medicare. And then we weren’t sure if they were going to be counting the number of requests for technical assistance as individuals. So one organization might ask for 20 TAs, does that up your number or is it the number of organizations that request regardless of how many requests for TA they make. And we thought maybe, going with our sense, that maybe a better phrasing of this measure would be the proportion of MCOs requesting technical assistance with MCH data. But a lot of it really depends on what was the significance and outcome that the state was looking for and our crystal ball wasn't clear on that. In regards to performance measure—
DONNA STROBINO: Well before we get to that one, does anybody
else have any comments? Actually this one struck me as sort of interesting because
I would have gone a completely different route. Because I was thinking about
when I did some work in Pennsylvania looking at the Medicaid expansion, the
Better Start Programs. One of the major problems they were having been we wanted
to access Medicaid data but we couldn't any longer. Because the Medicaid data
were now for the (inaudiable) service in the Medicaid office but for the managed
care organizations, and I think Pennsylvania was one of the early ones to get
a waiver, those data were no longer accessible from the Medicaid office. And
so I went the different route there thinking about can I actually access this
information. So I didn't know whether they were thinking in those terms or not
as well. So it was--interesting question.
Unidentified Speaker: It was an ambitious, certainly an ambitious area and quite
challenging.
DONNA STROBINO: Yes.
UNIDENTIFIED SPEAKER: To put together and we respect the entity that did put it together because it was certainly a good start.
DONNA STROBINO: Good.
UNIDENTIFIED SPEAKER: In terms of measure 10, teenage pregnancy rate for the girls? We felt that it was fairly clearly stated in measurable terms. Open to different interpretations? We thought not. And that it did not combine related or unrelated items and in terms of focusing on a unique outcome or process, we thought yes, it had. Our critiques were more around the numerator and the denominator that the numerator needed to have a definition of gestational age. What time period so that somebody from outside coming to look at it could apply the same definition. And that in the denominator we would drop the last two words, times and a thousand, and just leave it number of females aged 15 to 17 years of age. We had a bit of discussion about, sort of, the words describing the performance--are the performance measure, teen pregnancy rate for girls ages 15 to 17 debated back and forth whether we should talk about increasing, decreasing, or whatnot, having that in the title. The goal, we felt, could have been a little more specific by giving an amount that the state wanted to reduce the 15 to 17 pregnancy rates by. Otherwise, I think we started to--We were working on number three when the time was called so--But felt that this was a pretty good one with not a lot of tweaking needed.
DONNA STROBINO: Other comments? Cathy?
CATHY: But you do combine two very different data sources here and it's the issue, you're estimating--It's not clear whether you're--I guess they should be very clear because most Vital Records Departments do not have abortion data reported there. So if this state does, that makes it unique and so I would assume and, like you say, your gestational age or birth weight or weight, you know, are we talking late fetal deaths, you know, second week spontaneous abortions, what are you talking about I think that's critical. The other thing, at least as an issue, is depending on how good those estimates are, you're looking at a very small age group.
Unidentified Speaker: Yes. Right, right
.
CATHY: With a relatively, and we at least ourselves, have continued
problems with the census we have as a state group that does intercensal year
estimates. This is not a standard census five-year interval and so even the
question, like they do estimates 15 to 19. And then you get into the question;
well this is three of those five years. Do you take 60 percent of that or do
you assume, you know, are you getting the younger girls, or we get a lot of
older kids because they come to college. So particularly because it’s
such a sensitive issue, some discussion somewhere in a note or something else
about are you going to be able to come up an active denominator for every year
that you have your vital records?
UNIDENTIFIED SPEAKER: Well, but I think the point there to making is they don't indicate where they're going to get the denominator data.
CATHY: Well it's going to be estimated by their Bureau of Biometrics.
DONNA STROBINO: Well but it's estimated from what? Census data and repeated population surveys.
CATHY: Perhaps.
DONNA STROBINO: It didn't say, it wasn't--It didn't tell of the data source.
DONNA STROBINO: That's right. Right. It did not tell us the data source. It told us who was going to do the estimates, but they didn't tell us the data from which they were going to do the estimates. And if you got a sense of that you might feel a little more comfortable.
UNIDENTIFIED SPEAKER: I have to comment. In the sense that, you know, we're asking our own states (inaudible) each year and if you're going to explain the source of the denominator or the population estimates—
DONNA STROBINO: Right.
UNIDENTIFIED SPEAKER: Then put in the fact that one gets it from your Bureau of Biometrics, one assumes that they have a demographer that is generating it and is available. You know, I guess I'm going back to how much detail, do you think you need for a state to put into this? Now I could see if the state is looking at another indicator, a state needs to know if they want to make comparisons with another state, then they need to know how the numerator and denominator is applying with those comparisons. But, I mean, going through this and doing this every year, you know, to put the detail that you need for every state, for example, define how you define a denominator. Are you getting it from (inaudible)?
UNIDENTIFIED SPEAKER: Staff turnover is a big issue and I would argue it, that's very helpful for the next person who comes in who has to do that. And until we change the method, we don't have to change the definition.
UNIDENTIFIED SPEAKER: Exactly.
UNIDENTIFIED SPEAKER: Estimate is a critical word. I would
assume that vital records, and you're talking about births, you know, MCHB is,
sort of, unless there's something really unique about our vital records, I don’t
have to explain that but when it says estimated, I want to know how you estimated--a
little bit about how you estimated or how frequent.
Unidentified Speaker: That, no, and I appreciate what you're saying because—
UNIDENTIFIED SPEAKER: Do you want those states' whole internal procedure go to (inaudible).
DONNA STROBINO: No, no, no, no, no, no.
CATHY: You just want to know that there is one!
DONNA STROBINO: Yeah, I mean, I, you know, I'm a demographer so I wouldn't ever construct a (inaudible) unless I knew exactly where those data came from. And frankly when I write a grant proposal, if I don't tell details about how I'm going to collect the data, I'm not going to get funded. You know, I personally feel that you're only talking about, what is it, 18 national measures and seven to 10 state measures? I don't think it's inappropriate to ask people to be precise here. I mean, I appreciate where you're coming from but can't somebody pick up a phone and call the office and say, oh by the way, did you use census data to estimate that and were they 2000 census data or have you updated them? I don't think that's an inappropriate question to ask. But I appreciate where you're coming from, I mean I understand that.
UNIDENTIFIED SPEAKER: And I would say it's a learning tool. Having used the marvelous ness of MCHB having all of our grants up on the web is the ability to look at other states and their measures as we're trying to develop. And so it's an opportunity to do learning on a more self-paced basis rather than having to always make phone calls, saving those phone calls for more critical—
UNIDENTIFIED SPEAKER: I just wanted to (inaudible) it's really we just; you've already identified where, you know the area where somebody will produce the data. Those people in the Biometrics should be able to tell you, these are the data source. Because all they need to say is census data, vital records, is not to define the protocols, more or less.
DONNA STROBINO: Yeah but he has a good point in a sense that you may call that group and they won't give you, oh it was, you know, give you a two minute answer, they may give you the entire protocols of how they do it. And that's not useful for you necessarily.
UNIDENTIFIED SPEAKER: Exactly.
DONNA STROBINO: So it's a tradeoff on making sure that you've got it but it's not a detail
CATHY: I think also, one of the things that might, as you finish this sequence of these, because I know--I'm sure there's somebody else that's the competition but Massachusetts was well in the top five of number of notes on almost anything we put (inaudible). One reviewer made a point, actually from MC, you know, do you really need all this stuff? And we said yes, we do partially because internally the state people are constantly questioning, "What the hell did you mean by that?" And staff turnover in, there's sort of that tradition of, you know, the boss was saying, it just isn't enough for me. This last time when we got reviewed, actually a couple independent reviewers said that the notes were extremely helpful because as you read through those you really--Every time you thought of a question as you looked at somebody, like what do they mean.
It was there in the notes either to say we know this isn't quite exact, like on the equivalent off this biometrics. I put in the website address so that you can go and then they'll go through their whole thing. But I'm also saying this is where I got these dada from. And I agree, I agree because when I look at other states' things, it's really, you know, there's great ideas there but if I'm still not sure how you did it, I would just like to learn more about it. And you only have to write them once on these things. Once you change the measure, it's done.
DONNA STROBINO: And I guess, maybe, sort of, the message here is that I think the bureau appreciates the fact that this is a lot of work. But I guess to make sure that the data folks are part of the process. So that you have that kind of communication that you need so it's not too much extra work for you to be able to add in some of those notes or some of that additional information. Other comments or objections? All right. We're just going to--Jeff, do you have any comments? Great! So a few take home measures--measures--messages! I've been to measures today! Whatever measure is chosen it needs to be clear, reliable and valid. There is no question that that was a discussion point on the examples that we looked at. Jeff already said this but I think it's really important to emphasize this. This is state and national performance measures, should represent a balance of capacity, process and outcome measures. I'm always struck, in particular if I'm reading a study about intervention and the people are interested in looking at outcomes related to the intervention and they don’t collect information about the intervention.
How do you know how something happens in your state? Or happens when you make an intervention, unless you know what's being given, what's being done. That's really important so it's got to be a combination of all of those factors. Process and intermediate measures are acceptable and in some instances they're preferable. In some instances what you want to know about is what was done for or to your population or you do want to know about risk status. It's okay to be at the developmental level or the developmental stage for some of the performance measures. And that, in fact, if that's where you are, your performance measures should reflect that, it shouldn't reflect where you want to be after you've done the next step. So if you're trying to increase capacity, you want to say you're trying to increase capacity, you don't want to think about, okay I've increased capacity so this is what's going to happen. Development and the use of performance measures is an ongoing process. It's an ongoing process both in terms of what you do internally, but also as many of you indicated, it's an iterative process.
But if you promise something you need to follow up on it. And it's okay not to have done it, but you need to follow up on it and people need to know whether or not you did it. So it's important to at least follow up on your commitments and on your data objectives. Any other messages that people thought came out of the discussion today that they thought we should emphasize? You guys were great. And what I really liked about it is that you weren't afraid to bring a differing position as to what we were saying here. And I think we appreciate that people and states are at different stages and that some of this process when we're looking at specific content areas or we're thinking about the process in general, can be perceived as overwhelming when you're in the process of doing it. Cassie, do you have something else to say?