Dental Quality Alliance (DQA) – The Measurement Challenge

The Dental Quality Alliance (DQA) hosted its 2015 conference titled The Roles of Quality Measurement at the ADA Headquarters Building in Chicago on May 1-2, 2015. The goal of the Conference is to develop dental quality ambassadors who can help lead change within the profession.

I gave a presentation titled “Dissemination and Implementation – Routine Practice“ at the Conference. Please find my slides and the references here as PDF. Several speakers during the conference raised the “lack of diagnostic coding available in dental claims limits the ability to collect and report this type of data” as identified in the 2010 NQF report “Oral Health Performance Measurement Technical Report.” This topic will be explored further during the planned dental diagnostic terminology conference titled “Toward a Diagnosis-Driven Profession” which will be held in March 2016 in Los Angeles immediately following the annual conference of the International Association for Dental Research (IADR). If you want to receive further information about the conference please contact Dr. Elsbeth Kalenderian or myself.

About the DQA
In 2008, the Centers for Medicare and Medicaid Services proposed that the American Dental Association take the lead in establishing a Dental Quality Alliance to develop performance measures for oral health care. The DQA is an organization of major stakeholders in oral health care delivery that will use a collaborative approach to develop oral health care measures.

The DQA Guidebook succinctly states why self-evaluation is an important part of dentistry: “The dental profession is taking the lead in developing mechanisms for self evaluation. Self-evaluation will ensure that dentistry as a profession can provide evidence to the community at large that its members are responsible stewards of oral health. A culture of self-evaluation is the key to fostering the best healthcare for our patients ensuring transparency of health care quality and maintaining the credibility of the dental profession.”

However, there are huge barriers in dentistry when it comes to measuring health outcomes: “There is limited knowledge of true oral health outcomes, which occurs in part because dentistry does not have a tradition of formally reporting specific diagnoses or associating such diagnoses with specific services, (ref) especially through the claims process. Further, most dental practices and dental plans lack information systems capable of capturing and transmitting the information necessary for measurement (ref). Although retrospective claims data have many

University of North Carolina School of Dentistry prepares for implementation of Cloud-based electronic health record system developed by the UNC, University of Michigan and University of Pittsburgh with ICE Health Systems for Internet2’s NET+ service

An international collaboration of academic research centers, government agencies and industry discussed the initial implementation of the first Cloud-based electronic health record system for dental practice and education.

The pilot project, which will launch at the University of North Carolina School of Dentistry in 2014, is an effort to use next-generation technology to enhance dental record-keeping, education and research while maintaining patient privacy, confidentiality, security and legal standards.

For the last year, experts from UNC, the University of Michigan and the University of Pittsburgh have been developing the concept of a Cloud-based electronic health record for dentistry with representatives of Internet2 and Calgary, Canada- based ICE Health Systems.

The collaborators attended a week-long meeting at UNC last month to discuss final security assessments, including a third-party security audit, and to review the workflow analysis and stress testing that was conducted in preparation for the migration from UNC’s home-grown electronic record system to the new Cloud-based EHR, which will be offered via Internet2’s NET+ service.

During the meeting, UNC faculty, staff and students experienced the system first-hand in walkthroughs and demonstrations. Presentations explained the benefits of using a scalable electronic health record system for patient care, education and clinical research.

More information about the project and its developers is available at:


ICE Health Systems:

About Internet2

Internet2® is a member-owned advanced technology community founded by the nation’s leading higher education institutions in 1996. Internet2 provides a collaborative environment for U.S. research and education organizations to solve common technology challenges, and to develop innovative solutions in support of their educational, research, and community service missions.

Internet2 consists of 220 U.S. universities, 60 leading corporations, 70 government agencies, 38 regional and state education networks and more than 100 national research and education networking partners representing more than 50 countries. Internet2 offices are located in Ann Arbor, Mich.; Emeryville, Calif.; New York, NY; and Washington, DC.

About ICE Health Systems

ICE Health Systems is an innovative Electronic Health Record and Practice Management System whose development has been driven by a distinguished group of dentists and physicians representing a number of specialties in both university and private clinics. ICE Health Systems is a user-friendly practice management solution. Using cloud technology, this system supports the first multi-disciplinary practice management service providing integrated professional communications and continuous system enhancement.



Promoting Research by Standardization of Caries Risk Assessment

I just returned from the annual COHRI Winter Meeting which was held in Vancouver, Canada. During the meeting, I had a chance to talk to Dr. Joel White, Professor in the Division of Biomaterials and Bioengineering, Division of General Dentistry, Department of Preventive and Restorative Dental Sciences, School of Dentistry University of California, San Francisco about his leading role in COHRI’s CAMBRA workgroup which currently includes 57 members. The CAMBRA group was formed to standardize the caries risk assessment form in an electronic health record across dental institutions. His group also has developed a COHRI CRA Short form, which includes 6 questions to determine caries risk. This COHRI CRA Short form includes clinical decision support and automatically calculates caries risk based on the answers to the 6 questions.

Please watch the interview:

If you are interested in the fully operational electronic patient record form, just send me an e-mail to Joel at and he can have the COHRI CRA and COHRI Short Form loaded on your school’s test/train/live axiUm server.

Associate Dean, Office of Faculty Development and Information Management
Associate Professor, Dental Public Health, Center for Dental Informatics
School of Dental Medicine, University of Pittsburgh

Does dentistry really need more than one diagnostic vocabulary?

In case you had to guess, the answer is “no.” Recently, ran a story about the EZCode system and SNODENT titled “Diagnostic dental codes: Are we there yet?” Unfortunately, the answer to that question is also “no.”

The emergence of not one, but two, dental diagnostic vocabularies is troubling. First we have essentially none, then suddenly two. This reminds me of a recent quote by Doug Fridsma, the Director of the Office of Standards and Interoperability in the Office of the National Coordinator for Health IT. At an AMIA 2012 Annual Symposium panel discussion, he remarked: “Standards are like toothbrushes. Everyone has one, but nobody wants to use someone else’s.” I appreciated the dental analogy, but duplicate standards are no laughing matter. Dentistry is currently wasting a huge opportunity to create a novel, forward-looking approach to representing dental diagnoses. Unfortunately, we seem to be stuck somewhere between the Stone Age and the 19th century.

To understand why, we need to look back a few years. The ADA started working on the first incarnation of SNODENT, its diagnostic vocabulary, sometime in the 90s. The project took a while and SNODENT was supposed to be released in January 2000 (see “SNODENT to provide inclusive means of transmitting dental information,” ADA News, 30:9; 5/3/1999 [unfortunately, not available online]). When the ADA News asked then ADA President Dr. Tim Rose: “Will you be using SNODENT in your office?” he confidently answered: “I sure will.” Fast-forward to 2012: Yes, we are still waiting for SNODENT, now going into its second incarnation. (Actually,can you be reincarnated when you have never been born? Sounds like a Zen koan.)

In part, the EZCodes diagnostic vocabulary, a project of Harvard University School of Dental Medicine‘s Dr. Elsbeth Kalenderian, emerged as a reaction to this “Waiting for Godot” scenario. Normally, few people would care about competing dental diagnostic vocabularies, were it not for two important factors. One is the HITECH Act. Sometime in the future, the Department of Health and Human Services will anoint one dental diagnostic vocabulary as “the” standard for interoperability of diagnostic information in dentistry. The second is that the ADA is realizing quite a bit of non-dues revenue from licensing the CDT. (According to a recent conversation with an ADA staffer, a CDT license is about $11/year per customer of an electronic practice management system and $1,000/year per institutional site license. With over 90% of all dentists using a computer in their office, you can do the math.) So, it stands to reason that licensing a diagnostic vocabulary similarly might generate another nice chunk of cash for the ADA every year. As a result, Harvard and the ADA have been at war over their respective diagnostic vocabularies for quite some time. (Of course, if you only have the slightest inkling about the organizational psychology of both entities, you know this had to end up in a mudwrestling match. But, that is another story.)

So, what about the comparative merits of SNODENT and EZCodes? The DrBiscuspid article provides some basic information: SNODENT has about 7,000 terms, EZCodes about 1,300. EZCodes was developed by a working group of the Consortium for Oral Health Research and Informatics, mainly by merging several existing dental diagnostic vocabularies. SNODENT was developed through a somewhat opaque process that, to my knowledge, was never really published. Both vocabularies are currently free for researchers after signing a licensing agreement.

So, how well do these codes work? According to the DrBicuspid article, the EZCodes are currently being piloted in 17 dental schools and institutions located in the U.S. and Europe. There is a 2011 paper on the evaluation of the Z Codes, a major component of the EZCodes. SNODENT is rumored to be evaluated in a few dental schools, but a search for “SNODENT” in PubMed today only turned up the same three papers that have been there since 2006. The only other reference to a comparison between the two vocabularies alleges that “the ADA sees EZCodes as an ‘interface terminology’ useful for capturing health problems but not as a replacement for SNODENT in storing information in EHRs.”

So, what should we make of all of this? I fear that neither effort at developing a dental diagnostic vocabulary will produce a very satisfying result in the long term, unless some radical changes are made. Even worse, the tug-of-war and duplicate work consumes precious resources that dentistry, as a profession, can ill afford to waste. Here are a few relevant observations:

  • The world is, in general, moving away from top-down, bureaucratic approaches to developing and implementing standards. Why? Because they don’t work. The healthcare landscape is littered with ailing, moribund or just plain dead standardization efforts that consumed a lot of time and energy, and are essentially not used in practice. The more promising approaches are smaller, nimbler and less bureaucratic, and engage the  communities who care about and use the product from the very beginning.
  • Dentistry has a very successful and broadly used coding system, the Current Dental Terminology (CDT). The CDT has about 710 codes. Clinicians know most of the ones they use frequently by heart. Clearly, knowing codes by heart gets harder the larger a code set is – difficult with 1,300 codes and fairly impossible with 7,000. However, that is not an unsolvable problem. The entry interface for the codes in the electronic patient record simply must be smart enough to make choosing the right code easy for the user. This is a significant, but solvable, human-computer interaction design challenge.
  • Speaking of design: One adage in the design community is “fail early, fail often.” Bringing something as complex as a new diagnostic vocabulary online rarely works with a big-bang approach. It makes much more sense to focus on smaller pieces of the puzzle, get the bugs out, and then move on to developing the next bigger increment. (One clue for this is hidden in the Z Codes evaluation paper cited above: Over a period of one year, UCSF used only 93 [63%] of 147 Z codes.) Unfortunately, developing a dental diagnostic vocabulary in an incremental, iterative approach would require a level of collaboration, shared vision and coordination between dentists, informaticians, vocabulary and terminology specialists, and the dental IT industry, that is unlikely to materialize.
  • Coming back to the statement above that “the ADA sees EZCodes as an ‘interface terminology’ useful for capturing health problems but not as a replacement for SNODENT in storing information in EHRs,” we need to clear up a misunderstanding.  Separating work on various aspects of a vocabulary makes no sense. As Kent Spackman states in an authoritative paper on terminologies, ideally, interface and other terminologies should be derived from a common reference terminology because this “may allow different terminological efforts to focus on separate parts of the problem and to cooperate in solving the overall problem.” Given what we are witnessing, wishful thinking indeed!
  • Unfortunately, both vocabulary development efforts decided to stick with outdated models of representing classifications and terminologies. Over the long term, those approaches will be about as efficient and effective as the horse and buggy are for transportation today. The formalism for representing “things” for the foreseeable future are “ontologies,” which even JADA discovered in a 2010 editorial. For a number of reasons, ontologies are way more powerful for representing diagnoses, treatments and other concepts in healthcare than traditional approaches. (One thing they do very elegantly is to combine the terminology, information and inferencing models described by A. Rector in The interface between information, terminology, and inference models.) So, at this point, ontologies are the way to go in architecting vocabularies. The good thing is that you can largely reuse the work spent on creating vocabularies when you build ontologies, so not all past effort is wasted.
  • Developing and maintaining large vocabularies requires a lot of time and money. Very few organizations have the wherewithal to support this process. Here, again, we can take a lesson from the ontology world. Many ontologies are developed in a completely open process by their community of users. While development still must be organized and regulated, costs and effort are spread over a much larger number of individuals, groups and institutions. This has two benefits: (1) everyone who needs the ontology uses it and (2) no one has to ask how much it costs. One example: the Gene Ontology, one of the most successful ontologies ever created.

So, what will the future hold for dental diagnostic vocabularies? Given the current path, most likely mediocrity, tension, conflict, widespread dissatisfaction and little benefit. Not a pretty picture.



– Titus Schleyer, DMD, PhD

Assoc. Professor and Director, Center for Dental Informatics

P.S. In case you have not heard, an important pioneer of medical informatics, Dr. Homer Warner, passed away recently. Learn more about him in this obituary and video.

Integrating the old and new worlds: The example of the dental recall postcard

Integration of software, hardware and services is an important aspect of health information technology, as well as technology in general. When things are not well integrated, we notice: the automated blood pressure meter which doesn’t transmit its readings to the electronic health record; the DiagnoDent value that we have to write in the progress notes because there are no fields for it in the electronic dental record; or, the intraoral camera that does not automatically switch the imaging program to “capture” when we taken it out of the delivery tray. All these breaks in integration make us notice (and get annoyed about) technology. Technology gets in the way of us getting real work done. We wrote about some of these and other integration issues in a 2004 article in a JADA supplement. (Not to burst your bubble beforehand, but not much has changed since then.)

This situation is all the more reason to notice (and appreciate) instances of integration done well. I came across such an instance during a presentation about Dentrix’s eServices product suite the other day. It is not quite clear to me what exactly Dentrix eServices are. Dentrix’s Website on this is full of marketing-speak and thus less than useful. The gist of it seems to be anything where information gets sent around to process transactions (like appointment reminders and patient payments).

The great example of well-done integration came in the form of a postcard to remind a patient about an upcoming appointment.

Okay, so the front of the card is less than overwhelming. The interesting story is on the back.

The back of the card displays a two-dimensional barcode, also called a Quick Retrieval (QR) code, on the right-hand side. For those of you using smartphones, not a big deal. The great thing, though, is what happens when you scan the QR code: It brings up a window on your smartphone that lets you confirm your dental appointment right then and there. If I remember correctly, it also puts it on your calendar for good measure. (I was trying to get a slide with a picture of the app from Dentrix, but so far no luck.) If you don’t have a smartphone, you can use the Web address on the card to do the same thing on your computer.

The nice thing about this example of integration is how easy it bridges the hardcopy with the electronic world. Scan the card with your smart phone, push a few buttons and you can move on with your life – while not forgetting your dental appointment down the road. Wouldn’t it be great if technology worked this smoothly all the time?

All the best!


– Titus Schleyer, DMD, PhD

Assoc. Professor and Director, Center for Dental Informatics


How do we effectively get innovations in human-computer interaction into real-world software applications?

The other day, I got a Tweet from Ben Shneiderman at the University of Maryland about a human-computer interaction (HCI) innovation called Manylists. Ben is one of the world’s topmost HCI experts and one of my favorite Twitter leaders. Ben has an extremely high signal-to-noise ratio in his tweets, and if there was only one person I could follow on Twitter, he would be one of my top choices.

ManyLists is a product comparison tool that compares product features using spatial layouts with animated transitions. In simple terms, ManyLists arranges product features in a table in such a way that makes it easier to compare multiple products. The layout facilitates rapid scanning by the user, something I think we would all appreciate when buying a washing machine, a digital camera or junk food.

ManyLists looked similar to another tool developed in Ben’s lab, a medication reconciliation tool called Twinlist that I saw at a conference last year. Twinlist facilitates easy comparison/merging of a patient’s medication lists. For instance, a patient may be discharged from a hospital stay with a list of medications that may not be the same as she usually takes. Medication reconciliation, as it is typically done, is a relatively error-prone and effortful process. Twinlist does not fully automate the task, but provides clear advantages in helping the healthcare provider decide which medications to keep and which ones to drop.

Right after I got Ben’s tweet, I talked with Catherine Plaisant, an Associate Research Scientist who leads these projects. I was interested in the answers to two questions. The first one was what kind of HCI innovations Twinlist and ManyLists represented. Were they real breakthroughs or just incremental improvements? Catherine’s answer indicated that she thought they were somewhere in between. She pointed out that the applications really integrated several aspects of HCI done well: good graphic design, good choice of fonts and colors, and helpful animations to make  computational processes and their decomposition explicit.

The second question was more difficult to answer: How was she going to get these innovations into real-world software applications in an efficient and effective way? All over the world, programmers are busy creating medication reconciliation software. If they knew about her work, they could maybe improve on what she had done, instead of reinventing the wheel. Catherine did not have a good answer for that. Yes, the code and designs are openly available on request (just email plaisant at But, it is an unsolved mystery for how to get these innovations into the hands of developers better than we are able to today (which is to say, not very well).

A few years ago, I came across the Common User Interface, a project by Microsoft in Great Britain, that offers a fairly large set of well-designed and tested user interface components for electronic patient records. I have always dreamed of plugging a new patient record together from these components. That would be certainly less work than writing it from scratch!



– Titus Schleyer, DMD, PhD

Assoc. Professor and Director, Center for Dental Informatics

How can we improve the design of Electronic Dental Records (EDR)?

Those of us who develop systems for dental clinicians to use during patient care face a perennial quandary: How can we design systems that work like the intended users need (and want) them to? The way people usually try to answer this question goes something like this:

Programmer to dentist: “Tell me what the software should do.”

Dentist to programmer: “Well, I need it to do A, B and C. Also, we need to keep a record of D, E and F. … Tell you what. Here is a copy of the paper record we use – just make it do something like that.”

[3 weeks, months or years later] Programmer to dentist: “Here is the software I wrote for you.”

Dentist plays around with it for a while, then says: “That’s not what I needed …”

(A cartoon illustrating this phenomenon is here.)

Dissatisfaction among dental professionals with their electronic dental record (EDR) systems shows that there is a lot of work to do to close the gap between EDRs and the requirements of practice. One of my colleagues, Dr. Thankam Thyvalikakath, recently completed her PhD thesis titled “Designing clinical data presentation using cognitive task analysis methods” which has the potential to help narrow this gap.

I would like to talk a little bit about the basic informatics research that Thankam completed so ably. In order to find out how dentists review patient cases and make decisions, she conducted a relatively simple experiment. She asked 10 clinical practitioners individually to work through a set of three patient cases of low, medium and high complexity. The cases were comprehensively documented and presented in a standardized fashion. The experiments were audio- and video-recorded, and coded and analyzed. We then developed an EDR prototype, the DMDProject, to address some of the cognitive requirements which we elicited.

Here is an example:

As the figure shows, the ten study participants reviewed a patient case first by focusing on general patient information, medical history and the social history. They then proceeded to intraoral images and radiographs, followed by hard and soft tissue charts. Towards the end, they reviewed patient notes and refocused their attention on medical issues. The patient case was fairly complex, which was reflected in the participants’ intensive engagement with the patient record.

One of the key results of our analysis was that dental clinicians wanted an “overview” of patients whose record they reviewed for the first time. Strangely enough, very few EDRs offer this feature, which is relatively easily implemented using computer-based records but not paper-based ones. The resulting preliminary design implemented in the DMDProject looks like this:

In this design, the EDR shows a summary of the most relevant facts about Sarah Williams. It integrates demographic, insurance and clinical information. The last progress note is shown, as are scheduled procedures. The clinician also has easy access to the most recent images. (In the design, users can easily “drill down” to more detailed information from the Patient Overview.)

When we tested this design with users in several rounds of experiments, it was clear that it was a winner. Pretty much all clinicians commented very favorably on the usefulness of this view. So, we are hoping that more and more EDR vendors and developers adopt this design paradigm.

Thankam’s PhD thesis contains scores of other examples of how we elicited cognitive requirements of clinicians and then translated them into an appropriate design. Because it is somewhat hard to summarize a PhD thesis in a blog entry, I will come back to Thankam’s work in the future. (Sometime later this year, Thankam’s thesis will be available in Pitt’s D-Scholarship repository.)

All the best!


Titus Schleyer, DMD, PhD
Assoc. Professor and Director, Center for Dental Informatics

Dentists should use their patient data to …

… [your answer here].

In today’s blog posting, I am asking you to get creative. I would like to know what you think dentists should do with their patient data. (I explain why below.) So, the first thing I would like you to do before reading further is to complete the sentence above and post it as a comment in response to this blog posting. (If you have not posted on this blog before, I will have to approve your post, but I promise to do this quickly.)

So, now to the real question of this blog posting: Why am I asking you about what dentists should do with their patient data? The simple answer: Because I think we are not doing enough with them. 

Clearly, many of us use patient records to refresh our memory before an appointment, to answer a clinical question, to get a sense of what needs to be done next, and so on. So, our use of patient record data is primarily focused on supporting the care of individual patients. Nothing wrong with that. (Of course, we also use them to defend ourselves in lawsuits, but that is another story.) 

Beyond that, we also use patient records in the aggregate to some degree. For instance, we may check on groups of patients due for recall and send them a postcard or email to remind them. Or, identify patients who are overdue in completing their treatment plan, so we can call them to finish the care that they need. 

Beyond that … I don’t think we do much with our patient records. 

I think that needs to change. I think there is a lot of useful information locked away in our patient databases. For instance, they contain answers to questions like: Do resin restorations placed with the new bonding agent I started using last year have a higher incidence of postoperative sensitivity? In what kinds of patients does scaling and rootplaning not improve pocket depths? How long do crowns in my practice last? What patients are least likely to complete their treatment? Or, my favorite: What kind of dentist am I? 

Imagine that there was an easy way for individual dentists to ask these questions. Or, for that matter, a way to answer these questions using many dentists’ databases. This is one of the research projects we are working on (see “Data extraction using EDR in dental PBRN”).  

Our approach is designed to extract data from a variety of electronic dental record (EDR) systems in a standardized manner for purposes of quality assurance and research. At present, we are pilot-testing it with EagleSoft, but we are planning to add more systems in the future. 

The key philosophy of our approach is that we should be able to extract (for now, de-identified) patient data from a variety of EDRs in a standardized fashion to answer questions such as the ones listed above. This capability could be a highly valuable adjunct to the many clinical research studies being conducted in dentistry. Using data from practices, we should be able to conduct epidemiological, comparative effectiveness and other types of studies. 

Sounds like a good use of patient data to me. Do you agree? If so, what question(s) would you ask of your electronic dental record if you could? Looking forward to your responses!



Titus Schleyer, DMD, PhD
Assoc. Professor and Director, Center for Dental Informatics