Annual EMSC Grantee Meeting

TI Grant and PECARN Representatives

Beyond Peer-reviewed Journals:

Effective Dissemination of EMSC

June 20 – 22, 2006

MIKE TUNIK: Few more people are still coming in. So before we actually start with the material, I wanted to thank George Fulton for his leadership of the

center for peds emergency medicine. And I think without that much of this work would never have come to fruition. So our objectives for today are to list key

steps to assure dissemination of the results and the materials produced by EMSC targeted issue grants.

We're going to be describing three, at least three examples of EMSC projects that have effectively promoted regionally and nationally their dissemination

and distribution. And at the end we're also going to describe some ownership issues for EMSC‑funded grant products. And one of the other things as an ‑‑ it's not

a learning objective, it's a participation objective, is that after each of the five sections in the discussion, we are going to answer questions and ask for input from

any of the members of the PECARN targeted issue grantees who might have a great example of a dissemination or distribution process that we may not be aware of.

I think Dr. Sapien is here, and he had an example of web‑based education that was presented yesterday which is an example that we won't be focusing on.

So our first recommendation about dissemination process is to be inclusive and to try and identify and involve your end user very early in your project and that

will pay huge benefits.

MARSHA TREIBER: When we started back in the mid '90s to develop the teaching resource for instructors and pre-hospital pediatrics, which was for emergency medical technicians, we followed a two step process. First, we recruited a review board. We got representatives from every major national organization we could think of that had anything to do with emergency medical services, such as the state EMS directors and the training coordinators, and as you can see on the slide, emergency nurses, the NA EMT and the National Registry of EMTs, and American Academy of Pediatrics and American College of Surgeons, and the American College of Emergency Physicians.

And our aim was to create buy‑in to our product from the get‑go. And they actually did give us a lot of very valuable feedback every step of the way.

All of our products were never created in a vacuum. We didn't have a group of doctors sitting down saying what EMTs and paramedics had to know. We started

with people writing chapters. But then we took those chapters and we took them on the road. And Mike and I went all over the country, we went to urban, rural

and suburban settings, because we needed to make sure that we ‑‑ our information was relevant to the end users. So we had workshops for the first book, with emergency medical technician instructors, and then with the second book, the paramedic TRIP for paramedic instructors. And we brought them chapters, and they

had breakout groups, and we would leave them alone. They would evaluate the chapters, and we had people who would say: This chapter is no good. You should just rip it up and start again. And we did. And we had other people who said we don't do it this way in our region. And we said well tell us how you do it.

And so we were able to go all over the country so that we could try, as best we could, to be all things to all people. To make the books as relevant as we possibly could.

And when we first submitted our grant application, we got letters of support from all of the organizations who were going to give us a representative to review the chapters, because after the workshops, then the chapters went back to the review board. They looked it over, gave us more feedback and then Mike and I sat down, and every piece of feedback that we got from the end users to the review board was incorporated into the end product.

And we are very proud that 41 states, according to a survey that was done by the National Council of EMS Training Coordinators, indicated that 41 states had

adopted the basic TRIP as part of their teaching resource for pediatrics.

Our experience with Special Children's Outreach Education was similar but different. I think the CPEM people were a little bit more thoughtful and had a little more foresight about how they were going to distribute their materials. We originally in our first grant application, our aim was to create an education program for pre-hospital providers. The education of the management of children with special healthcare needs, and our goal was to educate people on a regional basis.

Our goal to teach DC fire and some of the surrounding counties in Maryland. We didn't put as much thought into spreading it any further and exactly how we were going to get buy‑in from the end user but we did a lot of the similar things as the CPEM folks.

The first thing I did was hire a nurse who was a paramedic who was a paramedic educator in the area. And we worked together on creating what we thought would be a really great education program, rolled it out locally for the EMS educators in DC and in some of the surrounding counties, and they gave us feedback. So it was a less formal type thing, but we really took what they said into consideration and created different iterations of the program as we spent more time going around to the various parts of the region and in DC.

So for us it was really a ground up type thing. We didn't seek letters of support from the national organizations or anything like that. And in fact in our second targeted issues grant when we decided we wanted an instructor program, we did try and talk to the various organizations, but we didn't have key contacts.

So we were not successful with that. But what happened with us, part of it just being tenacious, was that we spent a lot of time going out of our region giving the courses, and at that point decided that we needed to get formal feedback. And we probably had about a thousand surveys from EMTs and paramedics on what their needs were for the course and made changes that became part of the final program. So I really think it is important to solicit feedback from the people who are using your program. Especially prehospital folks, if that's your aim is to provide products for them, because they are very ‑‑ they have strong opinions about what it is they need and how they want to see their particular programs.

Let's see. What we did further, once we decided we had a program, we wanted to disseminate the program beyond the local area. So we decided to create a train the trainer program. And at that point we sought ‑‑ we solicited feedback with EMS leadership and EMS educators and held focus groups, and based on that we refined our program.

MIKE TUNIK: At this point I'd like to ask if anyone in the audience has any questions about some of the processes that were used to obtain end user involvement in either the TRIP programs or the Scope Program, since we were, I think, fairly brief as to what some of our methods were. And if anyone is progressing along a similar course, you know, some of the information as to what worked, what we learned from it, et cetera, might be useful.

Does anyone have any questions about the end user involvement approach or any details? Yes.

UNKNOWN SPEAKER: Would the TRIP program, alluded to having situations where we solicited feedback up front. You were able to take that and modify the material, did you have, give an example of any kind of significant conflicts you had where people had pretty strong and rooted beliefs in which way you should be going and you had to go to the right instead of the left and how you resolved that?

MIKE TUNIK: I think there were two stages of change, that we would have someone draft a chapter. We would then bring it to one of these workshops in a suburban, urban or rural region. We'd get feedback. Most of the feedback from the groups of instructors was not in conflict from the regions.

We would then incorporate that and send it to our review board. Every once in a while there would be some differences of opinion from some of our review board members. They were not usually diametrically opposed. And through some phone calls or just looking at other national standards, we were usually able to

adjudicate it.

Rare occasions we had to just choose, but I think that the fact that it happened rarely attested to the fact that the process was a pretty democratic one and inclusive.

MARSHA TREIBER: We were also careful in each chapter, whenever we were talking about different procedures, to always say in your region, with regional

protocols, so that we kind of let it ‑‑ we gave an out for people who did it differently, because we couldn't be all things to all people, as hard as we tried.

UNKNOWN SPEAKER: Can I ask a question, maybe you guys will answer this later. So you have to forgive me if it's a premature question. But how do you keep up with some of the things in your product?

MARSHA TREIBER: We do get to that.

MIKE TUNIK: I don't know if we'll have the best answer that will satisfy everyone but we'll get to it. Anyone else have any other questions about end user involvement? Contacts, for example, of the 50 state organizations that were involved with the national training coordinators?

I just want to make a comment. Marsha did a huge amount of phone leg work.

MARSHA TREIBER: That I'll talk about.

MIKE TUNIK: But it paid off. And does anyone have any examples of end user involvement that they feel they want to share with the rest of the, you know, target issue and PECARN audience here that we haven't, you know, discussed or alluded to?

Okay. So I think the next section was, you may have a great product but you want to make sure that people know about it and you get it out there. It's sort of like push e‑mail, if you will. And some ways that this might be done would be to plan on putting the material on a CD or now a larger video or moving image files, DVD.

Another option would be to load the material, if it's material such as the TRIP, which was mostly text and some drawn images, without a lot of graphics.

On a website that could then be downloaded from the website. And I think developing and maintaining the website certainly is something you should be thinking

about when you're writing your grant plan. And both of these, whether it's CD distribution or website distribution are, we found good ways to make your materials easily available. Now you may look at this picture if you recognize that this was the government warehouse at the end of Raider's Of the Lost Arc, the box in the middle is the arc of the covenant and it's going to go into the warehouse and no one is going to find it again. A pretty sort of wake‑up call to what happens with government products sometimes.

So our point is you don't want your EMSC targeted issue grant product to wind up in that warehouse. That is not, you know, the EMSC NRC storage site, but things can get lost. So unless you take a great deal of proactive responsibility for grant product that you're developing, that may be what happens to it. Great product, no one has it. So we're trying to encourage everyone to think about these things from the beginning of your grant, from the beginning of your grant application.

It's a comment.

UNKNOWN SPEAKER: I want to add one thing about that. I think this will be more emphasized in next year's targeted issue guidance. Under ‑‑ I guess it's 2002, the current administration, there's been less ability for programs to help distribute things that are developed by grant, as our grants are. And so that really grantees going to take, need to take more responsibility in their budget and planning how they're going to distribute this. And maybe they have those to be distributed through the EMSC center but rules have changed in terms of how things are developed under contract and distributed by us to have the person (inaudible) so that basically we can point people to the website or point people where they can get the resource from, but the grantees and the target issue guys more distribution plans, will have (inaudible) more the issues program, if they're developing a product that they're going to need to have a lot of distribution plan.

MIKE TUNIK: I'm glad you brought that up because that was why ‑‑ we had to make a big emotional impact. We're going to touch on that, and there are slides at the end of your handout that actually have that information. But forewarned is forearmed when you're writing a new grant application or even now if you want your grant product disseminated, you may want to change what your plans are if you have the budget for it, because the EMSC NRC is an official government website will not be able to distribute your stuff anymore.

MARSHA TREIBER: When we started to develop the first TRIP, we considered the idea very briefly of trying to get the book published at the end by some commercial company. And for us we really feared that if we did that it would really limit the distribution. So when we wrote our grant, we wrote it with our intention of distributing, producing and distributing the book ourselves.

And we ‑‑ so our main aim was just to get the information out there. So all of our products are available for free downloading off of our website. And as people would download information, they would register on our site. And when any changes happened to our books or to medical care, we could send an e‑mail out to people letting them know about it. And we did this for several years through a newsletter that we used to put on line, but unfortunately funding kind of occur tails that effort.

This is what our website looks like when you log on and Mr. Bear was the EMSC bear when we first developed the, very, very first website we had, he did this really cute little dance. And then when we changed to the new website, people really missed him so we brought him back.

So it's ‑‑ it's www.PCEM.org. And you have to click on him to get him to do his dance, and we're very proud to say that our materials have been downloaded by people from 60 countries around the world. So we've learned that from our registration.

UNKNOWN SPEAKER: You said that you try to get your commercial route, my question is if you don't use a commercial route, are you obligated pretty much to keep getting the grant updated over and over and over again.

MARSHA TREIBER: Yes, but one of the things that we did was after we gave out the initial, the initial load of disks for distribution, we started selling them ourselves. And the money that we sold the disks for went into a fund that created a pot to reproduce the book. And, yes, we did have to get an additional grant to do a major overhaul. And as a matter of fact, we've just completed the manuscript for the next version of the basic TRIP.

UNKNOWN SPEAKER: Let me ask a very important question. A well‑selling textbook sells about 3,000 copies. We've distributed, what, now, 25,000? 25,000 copies nationally by (inaudible) medical. So it's true if you go to a book publisher they will update it but it's a trade‑off between the penetration versus updating.

Updating is a challenge, because it is true at this point you're still looking for funding (inaudible).

UNKNOWN SPEAKER: Marsha, how much money are we talking about? Can you give us an idea of putting this on your website and then maintaining it on an annual basis, what monthly is the cost?

MARSHA TREIBER: Right now it's hard to say because when we began we found a web master who had an 18 month old who, when I talked to him on the phone I could hear the kid like gurgling in the background. So he was a real ‑‑ he was dedicated to our project. So the amount of money that he charged versus when we tried different commercial companies was so different that I couldn't even begin to tell you what it would cost right now.

And our site has already been developed and now we work for the NYU School of Medicine and we've just migrated the site over to NYU. So it doesn't cost us anything to maintain at this point. I mean just you know other than keeping up our domain name.

UNKNOWN SPEAKER: You have people around now who know how to do the programming so ‑‑

MIKE TUNIK: I have a comment in that.

The difference between a high end commercial group that was producing websites and the approximate amount that it might cost someone who's knowledgeable about it and capable, I'd find someone who has done website work that you could look at, was tenfold. It's a 5,000 versus $50,000. So it does pay as with all endeavors to shop around a little bit. Obviously $50,000 is going to break the bank of one year of a targeted issue grant. But five or ten is certainly within the budgetary ballpark.

MARSHA TREIBER: We were lucky, at the time we were able to produce the CDs, once the book was written and we got through that whole process, which took several years, we were able to get the disks produced for about $1 a piece. But the real key to it is getting it out there and distributing it. So for the first book I personally called every state director in the country.

He didn't know who I was or he or she didn't know who I was, but I was persistent and I got them to give me a name of someone who would be willing to distribute the book for us. And we told them that we would be giving them 100 CDs to distribute to their training centers, and they were allowed to reproduce it and download it and print it. And we actually did print several, a few thousand copies of the book, but it was an 850 page book that was in a loose‑leaf binder form that probably could have killed a small dog and people really weren't all that interested in the print version. So we don't even bother with that anymore.

And so the first ‑‑ my first effort really took a lot of effort, but when the paramedic TRIP came out, I sent, I talked to Beth Armstrong, who is the executive director of the state EMS directors, and I said could you please put a notice out on the list serve letting the state directors know that the paramedic TRIP is going to come out, and to ‑‑ I composed something that I needed. And for the name of a person who would be designated as the person to distribute it, and at that point things were getting more expensive and we distributed 50 copies to each state. But within 24 hours I had responses from 38 people. So I'd say that was quite a difference from the several weeks that I spent trying to hunt people down in the beginning.

And that was a major, major coup for us that we were able to do that. And I also attended many meetings of this organizations and at the state director's meeting I was able to announce the fact that we had in process the child abuse and neglect teaching resource in progress, and so they knew it was coming, so then, again, when I put out the notice on the list serve, people wanted it.

And what we did do was we sold additional ‑‑ everybody got, each state got 100 copies of the first TRIP and they got 50 copies each of the second TRIP and the child abuse book. And we sold them subsequently any more copies that they wanted we sold for $10 a piece and have been able to do reprints of ‑‑ reproductions of the disks with the money we earned from selling them after.

And we've been able to get, as George said, we've been able to get thousands and thousands and thousands of copies out there. And this is just the disks, this doesn't count the people who have downloaded it from the website.

UNKNOWN SPEAKER: The other part of that is the large number of times I seal people using our graphics in their talks, because it's a library of images.

MARSHA TREIBER: Yes.

TERRY ADIRIM: We use a lot of them in SCOPE. They're good. We got permission from your artists.

MIKE TUNIK: Some of the websites, we see they say CPEM on the bottom and some of them don't. And of course Terry asked about it and we said sure.

TERRY ADIRIM: We did some similar things. Again we didn't earmark any money for website but prehospital providers love the technology, and they like websites. They like DVDs. They don't like the print versions of things. So the cheapest thing for us to do is to have some pages off of the hospital's website, and of course we were discussing this earlier, but you need to make it very accessible if you're using your hospital's website, because a lot of times it's embedded in the website and it's very difficult for people to find.

We really wanted to disseminate the program. So we decided that the best way to do that is to train people to give our program. So that was the route that we chose to do. Because we couldn't go, you know, all over the place to teach the program. So we needed to get people to do that. So that's when we created a train the trainer program, and we had a very willing end users here in DC that we trained to use the program. So they were our feedback group.

Another thing we did was prior to, I guess, this year, you know, we made sure that our products were available through the EMSC website and in fact our manuals are available by hyperlink off the website. But the route we took as opposed to CPEM we decided to have our materials more formally published through a publisher. And Jones and Bartlett they publish PEP and we thought that would be a great way to get our materials out because then they advertise. They go to meetings. They advertise your product.

And so we did that. Our instructor's manual, which is the textbook was published last fall, and we have our instructor tool kit which would be published this fall coming up. So we do all the updating and the refinement and whatever it is they want us to do. But they actually put it together and put it in the kit and they're going to advertise it and they advertise it in all kinds of websites. So that was the route that we took.

The other thing that's great about this is that they're going to create a website just like PEP so it doesn't cost us anything they're undertaking that cost. So we thought this was a benefit for our program. So...

MIKE TUNIK: I think we've sort of covered this a little bit. But not only identifying the end user, but maintaining contact with the end users was also instrumental in CPEM's distribution.

So Marsha is going to talk a little bit more about that.

MARSHA TREIBER: As we discussed before, we created this huge network, and what I've continued to do is go to meetings every year and meet the new people and keep our product, our products in their faces, so to speak. We've been very, very grateful to the EMSC program for their incredible support and the development of all of our products, and we were able to do secure some funding. Once the heart association came out with their new guidelines, we were able to get a contract to revise our first book, because it was published in '97, '98. So once it goes through government clearance, because our next book will go through government clearance, so we're expecting the new basic life support TRIP to come out sometime next year. And we're excited about that.

Someone asked before how to keep things current. The website, the plan of the website, to quote our fearless leader George Fulton, was to keep the document as a living breathing document with each book by using the website as a source of information for updating changes in medical practice. And we did have in one of our newsletters we had a very interesting debate about intubation and whether children should be intubated or not. And that's been a controversy for that's ongoing. But it's ‑‑ the website has been a very useful tool for that.

So we did think about trying to get it published after our books got very popular. But at that point the publishers we spoke to weren't really that interested because we had given so much of the product away for free that there really wasn't all that much in it for them. So we feel that the network we created is ongoing, and that's what we're going to stick with.

So does anyone have any questions about the approaches to gaining regional and national distribution and maintaining contact with some of the end users or the organizations that we've just discussed? Again, details that might help with a similar approach or with a TI project that you might be working on or planning to apply for?

Okay. I think the next piece that is very helpful in disseminating information is to hold national or regional meetings. There has been in the past some funding for regional meetings regarding EMSC issues. And I certainly hope that that funding will continue, because that may be a way even outside of the targeted issue grant that you might be able to inform groups that would be very interested in your product.

Things that could be done at a meeting which we did do with the child abuse and neglect resource was to bring ‑‑ I'm sorry, let me go back. Is to bring different groups together to develop a consensus about what your educational or other product might need to have in it. They might also be conduits for the development process. You might find people who would write or edit sections of the material, and you certainly could disseminate information through this type of meeting.

Now ‑‑ Marsha, do you want to hold up the can since we have the wrong picture on there.

So the child abuse and neglect targeted issue grant really was focused on doing three things: One was to assess what EMTs across the country knew about child abuse and neglect, their educational experience, and their confidence in managing these children and what they desired in terms of other education.

So one piece to this grant was to collect data on a national basis in collaboration with the national registry of EMTs survey process and with NEDARC.

That information or at least a pilot of that information was presented at national consensus conference meeting that brought child protection groups EMS groups and EMS for children groups together, and it's probably the first time that the child protection groups and the EMS and EMSC groups actually sat down to talk about child protection issues that crossed from nursing or doctor or hospital‑based assessments to the prehospital phase.

So the participation, the participants in this national conference were given pilot information based on the knowledge, education and practice of child abuse and neglect management for EMTs and paramedics across the U.S. And therefore we were able to base recommendations on our educational product on evidence‑based rather than on well this is what we think we ought to be doing.

So the meeting did result in a consensus group of recommendations.

Next slide. And we were able to publish those recommendations in the three journals simultaneously, which really looked at the knowledge and attitude assessment and education and prehospital personnel and child abuse and neglect. The data was actually presented recently as an abstract of PAS and the manuscript of this data is currently being worked on. But to get back to the focus, holding that national meeting was useful because the groups all got together and agreed on something, and were willing to put those agreements down in essentially a meeting outcome.

We were able to base our educational piece on the recommendation and all of it was based on some evidence. And this was all done within a single targeted issue grant budget over three years. So I'll break my arm patting ourselves but I think it was a pretty good effort to get data, hold the meeting and produce a resource in one TI grant.

TERRY ADIRIM: We also did as I mentioned a bit of a needs analysis, every time we went out to teach the course we assessed EMTs and paramedics what they felt they needed for education about children with special needs and what their level of comfort is taking care of kids and their attitudes. And we did this. We went to about ten states and we got several hundred surveys that we analyzed and we presented the findings, these needs analysis. Both of the EMTs and paramedics but we also did a survey of parents, too, Washington DC of parents to find out what their needs were and what their expectations were of prehospital providers, and we also presented these at national meetings.

The other thing that we did is we had two publications in the Journal of Emergency Medical Services. That's a publication that's aimed at the prehospital environment. So we thought that would be a good way to disseminate what it is that we were doing to the end users.

The other thing that we did is we continued wherever we were asked to give scope courses. We went to a variety of state EMS conferences and provided train the trainer courses, and in fact there's some states, such as Ohio, where there's several sites that give scope courses. We continue to do it regionally and as I mentioned before we trained all DC fire and EMS.

MIKE TUNIK: We'd like to ask if anyone has any questions about what it took to pull a national or regional meetings together with the child abuse or neglect or the scope resource that we've mentioned, budgetary issues, logistics whatever you might be thinking about if you're going to try to do this yourself.

MARSHA TREIBER: One logistical thing I just realized on our slides, the incredible support that the EMS National Resource Center and NEDARC provide that people should really consider tapping into, because whether we did our child protection, the child abuse and neglect survey, we, Lenora Olson is sitting back there, and she was a big help in helping us analyze the data and I think that there are resources around that you can use that people just don't think of tapping into. And I just wanted to again say that the EMSC program is incredibly supportive of our efforts. And there are a lot of people that Jane Ball said this morning are ready to give you the technical assistance you might need to answer any questions that you might think of when you leave out of here.

TERRY ADIRIM: Sometimes with cost, like for us at SCOPE, we wanted to get this program out there. We did a lot of teaching and going out at cost. We didn't see it as a profit center. We were just so eager to get it out that we just asked for what our expenses were. So I know you all charge ‑‑

UNKNOWN SPEAKER: I have a question for national and regional meetings, what kind of suggestions or recommendations would you have for helping people to cope with the press, like the media, and how that might help in terms of broader assemblage, because often at some of these national meetings, specifically national meetings they tend to be conferences or reporters, I'm thinking of (inaudible) but any suggestions for how to tap in?

MIKE TUNIK: I have to say I don't believe there was heavy press presence at the national blue ribbon panel meeting for the child abuse and neglect resource. But I think what I've heard some people have done is to call, you know, local media, and basically tell them that there's going to be a national meeting of 12, 15 national organizations to discuss consensus on how to assess and manage child maltreatment from a prehospital basis, and you know if they come they come, if they don't, I'm not sure if there's a way to make them go there. But I think informing the media can be helpful. And if you do have regional and national presence and organizational presence at the meeting, and it is a hot button topic, I think that always helps. You're likely to get press presence.

UNKNOWN SPEAKER: I would say at the national organizations they have working people, and they have key contacts. I would explore that to inform people about the new product or paper that was accepted, that was going to be released in a certain journal, you would want to kind of spread the word.

TERRY ADIRIM: You can ask them for the contents because you could for free send out press releases because it's not like it costs anything.

UNKNOWN SPEAKER: Thank you, there were two things I wanted to add for national meetings, for the national organizations, sometimes they have subcommittees that you can, they're happy to have you join them. I'm on the education committee and the EC committee and the disaster preparedness committee for the state directors. So if you have an area of interest, you can try to be part of that organization's structure. That's a real nice way to really get your product out there. But another thing is that the EMSC has quick news that they send out to their mailing list all the time so that if you ‑‑

UNKNOWN SPEAKER: Every Tuesday night.

MARSHA TREIBER: Every Tuesday night.

UNKNOWN SPEAKER: Is that green tree's only? Quick list? Is it open?

MARSHA TREIBER: I don't know who it gets mailed to. But I know that we provide, any new publication that would come out that's not necessarily EMS supported but of EMSC interest.

UNKNOWN SPEAKER: In that vein I do the list serve which is completely open to anyone and I'll take your e‑mail and I'll add you on. It's not just open to grantees I get it out to the products and research and data are always welcome.

UNKNOWN SPEAKER: I realize that the focus of this presentation is on the (inaudible) and we talked a little bit about the evaluation yesterday. But I'm wondering whether what role you see for having evaluation information, product evaluation information available in order to promote the product. In other words, if I'm a consumer, why should I choose to use your product, what evidence do I have that it's going to accomplish what it's supposed to accomplish?

MIKE TUNIK: I think that's a good question. I believe that there's been, you know, certain probably political issues about having EMSC products reviewed and scored, which I think that's where you know you may be going. So that if there are five or six products that are aimed at a prehospital provider at a BS level for general pediatric care, which one should, you know, an organization choose as the product that they might go with.

And I think partly because one person, you know, scoring the product may have a particular perspective on it.

It might not be the same opinion as an individual who, let's say, someone works in EMS someone is a physician, someone else is an emergency nurse. You might wind up with I think a different perspective unless you have a big panel scoring every one of these products, and if you've looked at the EMSC product list, just the task of having people really take a close look at every one of these products, six or eight people on a panel and scoring them I think would be a pretty big and probably costly undertaking. So it hasn't happened yet.

I do know that people who work with EMSC have been asked to look at products that may be aging to see if they should still be maintained.

But the only thing I can suggest is to talk to someone at the NRC and looking at the list of products under child abuse or children with special healthcare needs, and if you identify four or five products that you might be educating a large group of physicians or nurses or EMS providers with, I would personally get a copy of each one of them and make sure that you're happy with, you know, the product that you choose rather than having a rating system. Because I've been dissatisfied with rating systems on the web.

MARSHA TREIBER: That's not really ‑‑ it's not formal evaluations it's subjective. What you're talking about more is objective.

UNKNOWN SPEAKER: Yeah. And textbooks to that standard either. So I guess the question is what is the standard, and I'm looking forward to issues that are coming up in educational curriculum where we're being held to talk about not the quality of the material, but really the quality of the output. In other words, what is the end user learning from using this material.

And the question gets back to the issues of value of education intervention: What is our responsibility as producers of these materials to evaluate them and to include the results of that evaluation in a sales pitch, if you will.

If you were talking about the sales pitch, or you were talking ‑‑ you haven't really talked about developing evidence that your project works.

MIKE TUNIK: I think it's a very difficult question, because when you have a relatively limited budget in three years and you spend probably a good chunk of that. I mean every time I look at the amount of work involved in one of the TRIP products, the SCOPE product, listening as Dr. Sapien testified how long it took to put the website and the educational material together, you're really chewing up most of a TSI grant just to develop something.

And unless you are working on another grant or extremely facile in getting your product together, you may not have a big chunk of time and effort to actually produce a high quality outcomes assessment of this education, which is what you're looking at.

I do think there's going to be a much stronger focus on outcomes in the targeted issue grant request for proposals in the future, because the EMSC program is being asked to come up with evidence that what we're doing is working. If you say you're developing a curriculum and that your outcome is wide distribution, and you can prove your distribution, I think that's not an unreasonable end point.

But if you're going to prove that people who use the educational product learn something, then I think you're going to have to have a fairly rigorous research methodology and large enough sample to demonstrate it. That might be the focus of a targeted issue grant.

MARSHA TREIBER: We actually put in a targeted issues grant proposal the last go‑around to, to try to determine the presence of all these products that EMSC program had funded over the last 20 years, but the grant review panel that there were other more important topics at that point in time.

UNKNOWN SPEAKER: A question popped to my mind they've already asked. The part two is looking at yesterday's presentations (inaudible) as a survey, as an instrument that we can you use by (inaudible) my question is two parts. One was how easy it because you've already done survey (inaudible) to access the complete hospital. And what was the response rate? You really didn't get a response rated, you would be able to get them meaningful results.

Part two is: Could we use that survey basically as we have disseminate it and see how it affected, excuse me, the performance or knowledge after it was not ‑‑

UNKNOWN PERSON: I think the question you asked was talking a little bit about how we did the survey. We were thinking of actually surveying an EMT's database of people who were numbers, maybe looking at other EMS organizations and although we might get a fairly large end, it wasn't necessarily a nationally representative sample, which we were trying to achieve. With 600,000 providers, with not every one of the providers names or emails or addresses in one place, it actually, if you're looking at it nationally, even large region assessment like that, it can be difficult.

So what we turn to is a group of prehospital providers that have essentially signed up to be members of the National Registry of EMTs. I do think it may be somewhat of a selected group of prehospital providers, but they basically had a very large membership and they basically surveyed their members once or twice a year, so that they had mailing lists; they produced surveys. They had scan trons to read the surveys. And we said if we work with you and provide you some resources, would you work with us and disseminate our survey. And they said yes. That's the way we went, was to use an existing survey system which is probably national representative as people have gotten so far with prehospital providers.

Survey response rate of the ‑‑ there are probably tens of thousands of people who are nationally registered, but they basically have a random group with an over sampling of minority and ethnicities that they surveyed. They surveyed about 4,000 people. We got about a 45% response rate, which is typical for their survey mechanism. Not bad for surveys, but certainly not 60, 70%. Given the limitations of the grant time and money we said, well, we're going to go with a nationally accepted survey mechanism in that response rate.

Was it meaning ‑‑ something about whether the data we got back were meaningful. I think if you believe that the survey mechanism is nationally representative, then for national sample, the answer is, yeah, we think it is. But whether you, if you have like a region that you trained as opposed to a national dissemination ‑‑

UNKNOWN SPEAKER: I don't think my question was meaningful. My question was: Can we use the same survey mechanism to evaluate the products which is responsible, if they were passed, we don't really know what this product does. And what was the impact of you on the provider.

MIKE TUNIK: If you're looking to see if there was a knowledge gain, a skill gain, an attitudinal improvement that this type of mail survey is not going to get you there. You would have to do maybe a random selection of hundreds of people out of thousands that you trained to see if you're actually getting what you think you are getting out of the education.

You may be able to determine if they know about your product, if have used or experienced using your product in their education, that is possible. In terms of whether you could use this mechanism, since this is essentially proprietary to the National Registry of EMTs, you would need to approach them as a grantee and say can we negotiate, you know, using your survey mechanism mechanic for, you know, remuneration because there's some not insignificant costs to doing an extra survey.

Other questions? Or how about other comments or examples about national or regional meetings that you know had an effective focus or twist to them?