AMSA Pharm-free Scorecard 2014

Methodology

  1. Methods
  2. Rating system
  3. Determination of letter grade
  4. Reassessment
  5. Assessment Instrument 

I. Methods

Each year, all U.S. schools of allopathic and osteopathic medicine are invited to submit policies to the AMSA Scorecard for assessment. Emails are sent to appropriate contacts at these schools (deans, conflict of interest officers, etc. as established by previous year's contact at each school) explaining the Scorecard project and requesting the submission of their policies. The email also contains a check-list identifying relevant policy domains and requesting permission to make policies public, as well as a submission guidance document.

In prior years, schools that did not submit policies after multiple attempts at contact were assumed to be unchanged from the prior year and retained an unchanged assessment. This year, however, we significantly revised the rating system (see section below). Therefore, carrying forward last year’s grade based on a different scorecard would not have been appropriate.

For those schools that did not submit policies this year (approximately half of all schools), we searched for conflict-of-interest policies online. In order to maximize our ability to retrieve all policies, we developed a systematic search process. This included using a list of search terms (such as “conflict of interest,” “industry interactions,” and “vendor policies”), searching the current AMSA site for any publically posted policies, checking the Institute for Medicine as a Profession (IMAP) website, which publishes a database of publically available policy documents, and, when we could not find information on particular domains, we emailed and called compliance officers to ask about specific policies.

In order to score the policies, we created a scoring codebook and provided formal training for analysts in its use.  To assess inter-rater reliability, the four analysts scored five training policies, and achieved an inter-rater reliability of 74 to 86%. Unlike prior years, policies were not redacted for blinded assessing. We made this change for three reasons: there was no evidence that this redaction process reduced bias; the process was labor-intensive; and redacting policies made them more difficult for analysts to read and to interpret.  Once all policies were scored, the senior analyst did a final check of all policies to ensure that the scores for domains were adequately documented and were accurate. 

II. Rating system

This year marked a significant change in the rating system. An AMSA scorecard methodology working group reviewed the literature on conflicts of interest, including the recent recommendations published by the Pew Task Force on Medical Conflicts of Interest[i]. As a result of this review, the number of domains was increased from 11 to 14 for medical schools, and 16 for teaching hospitals. 

This scoring instrument focuses on conflict-of-interest policies directly related to industry marketing and education. While not addressed here, academic medical centers should also have robust policies to ensure the integrity of basic and clinical research.

Policies are rated on each of the domains listed below, using the following general format.

3 = Model policy
2 = Good progress toward model policy
1 = Policy is absent or unlikely to have a substantial effect on behavior

Policies are rated on how closely they approach the goals, recognizing that there may be many approaches to a single goal.

“Industry” refers to the pharmaceutical and medical device industries.

Policy Domains:

  1. Gifts from industry
  2. Meals from industry
  3. Industry-sponsored promotional speaking relationships
  4. Industry support of ACCME-accredited CME
  5. Attendance of industry-sponsored promotional events
  6. Industry-funded scholarships and awards
  7. Ghostwriting and honorary authorship
  8. Consulting and advising relationships
  9. Access of pharmaceutical sales representatives
  10. Access of medical device representatives
  11. Conflict of interest disclosure
  12. Existence of an adequate conflict-of-interest medical school curriculum
  13. Extension of COI policies to adjunct/courtesy faculty and affiliated hospitals/clinics
  14. Enforcement and sanctions of policies

The following two domains were not used for medical schools, but will be used to assess policies of teaching hospitals.

15. Pharmaceutical Samples
16. Purchasing & Formularies

1. Gifts

Background: Numerous published studies demonstrate that small and large gifts play a role in influencing prescribing decisions, which directly affect patients. Medical personnel consistently underestimate the extent to which they personally are influenced.

3 = All gifts funded by industry are prohibited, regardless of nature or value.

2 = Gifts allowed but only if value ≤$10 or if the gifts are limited to educational items for physicians such as textbooks.

1 = Gifts allowed of value above $10 or no restriction specified.

2. Meals

Background: Like gifts, studies have shown that free meals can influence prescribing decisions.

3 = No industry-funded meals of any nature or value allowed.

2 = Meals allowed but only if:

(a) the value of the meal is ≤ $10,
(b) or when provided at industry-funded accredited CME events,
(c) or when provided on-site as part of an indirect grant from industry

1 = Meals allowed of value above $10 or no restrictions specified.

3. Industry-funded promotional speaking relationships

Background: Research relationships with industry may entail beneficial public presentations and speeches by individual researchers. However, industry also uses academic physicians to support marketing goals by identifying and cultivating speakers who give a positive message about the drug in question. Such ongoing relationships, sometimes called “speakers bureaus,” compromise the academic integrity of the institution.

3 = Policy effectively prevents faculty from being paid by industry to do promotional speaking, or to be on industry-funded speakers’ bureaus. An effective policy may ensure this in a variety of ways, but for the purposes the Scorecard, the policy must explicitly state the following two criteria:

(a) The talk is not promotional in nature, but purely educational; and

(b) Industry has no role in determining or approving presentation content.

2 = Industry-funded speaking relationships are regulated but with less stringent limits on content control, compensation, etc.

1 =No limitations on industry-funded speaking or no policy.

4. Industry support of ACCME-accredited CME

Background: Studies have shown that industry funding of continuing medical education programs tends to bias topic choices and content in favor of the sponsors’ products and therapeutic areas. Major medical organizations, such as the Institute of Medicine, the Association of American Medical Colleges, and the American Medical Association’s Council for Ethical and Judicial Affairs, have called for elimination or severe restriction of industry funding for CME.

3 = Policy states that industry funding is not accepted for the support of accredited CME courses except in certain clearly defined circumstances. Examples of permitted exceptions must be defined explicitly and may include:

  • The course would be prohibitively expensive/unaffordable to physicians without industry funding, or
  • Industry funding is accepted via a central, undesignated, blinded pool of funds with the central office having sole discretion on how the funds are to be used. For purposes of the Scorecard, a central, blinded pool is defined as such: a company cannot suggest or stipulate the specific course or academic program (i.e. department) for which the funding is awarded.

2 = Commercial support accepted, but at least one measure is in place to prevent promotional content (in addition to ACCME accreditation), such as:

  • Requiring more than one sponsor for any event,
  • Requiring that departments not be allowed to make a profit from programs (i.e., funds must be used for educational program only)
  • Requiring that physician participants cover some of the cost of the program (i.e. they may be required to pay for their own meal),
  • Requiring centralized management of CME programming (i.e., industry can award funds for a specific course but all funds must go first to a centralized CME Office, which is responsible for compliance with CME regulations and which disburses funds to departments for CME courses; this is not as stringent as a blinded pool of funds, which does not allow companies to designate funds for specific courses), or
  • Some other robust regulation or oversight of commercial influence

1=Industry support can be accepted with the only stipulation being that courses must follow ACCME criteria.

5. Attendance of industry-sponsored promotional programs

Background: Industry promotional programs generally provide educational content, but are created with the ultimate goal of increasing physician use of the promoted drug or device. Such programs are philosophically inconsistent with the educational mission of medical schools, which is to instruct broadly on the optimal care of patients.

3= Faculty, students, and trainees are prohibited or discouraged from attending industry-sponsored promotional events. Attendees cannot accept industry reimbursement for travel or other remuneration.

2=Attendance allowed (without any statement discouraging attendance) but attendees cannot accept industry reimbursement for travel or other remuneration.

1=No restrictions or no policy.

6. Industry-funded scholarships and awards

Background: Industry frequently provides funding to support students and trainees’ attendance at medical conferences. Allowing industry to subsidize educational opportunities risks persuading trainees to favor the funding company’s products. Policies that prevent companies from directly selecting recipients do not prevent students’ knowledge of the identity of the funding company, which in turn creates the same opportunity for companies to engender good will with students and trainees.

3=Industry support for residents and medical students to attend conferences or trainings is prohibited.      

2=Industry support to attend conferences or trainings is allowed, but there are one or more safeguards in place to ensure the funds are not used by the company to establish a marketing relationship with the trainee.  An example of such a measure may include selection of recipients through a competitive process managed by the dean or university administration. 

1=Industry support is allowed without stipulations or there is no policy.

7. Ghostwriting and honorary authorship

Background: Pharmaceutical and device companies often hire medical writing companies to ghostwrite articles that are subsequently published in the medical literature under the names of academics—termed “honorary authors”— who may have had little or no involvement in the research or in the writing. Physicians rely on the information they read in journal articles to make prescribing decisions, and they should be able to trust that any recommendations made reflect the research and opinions of the authors and not the hidden influence of writers hired by industry.

3=Industry-funded ghostwriting and honorary authorship are strictly prohibited.

2=The practice is discouraged, but not prohibited. 

1=No policy.

8. Consulting and advising relationships

Background:  While consultation with industry is crucial in drug and device discovery and refinement, some consulting agreements are explicitly focused on marketing. Such consultations vary in topic and scope. These types of activities place a physician in a promotional role for a product, which, like serving on industry-sponsored speakers' bureaus, is not consistent with the academic mission of providing unbiased medical education.

3= Policy specifies that consulting or advising relationships for purely commercial or marketing purposes are prohibited or actively discouraged. Consulting or advising relationships for research and scientific activities are allowed without prohibition or discouragement.  The policy also requires at least one of the following:

  • Prior review of the activity
  • Clear, legitimate deliverables spelled out in the contract
  • Fee is at fair market value 

2=Policy allows all consulting and advising relationships (research, scientific activities, and commercial and marketing consulting relationships are all allowed) but requires at least one of the following:

  • Prior review of the activity
  • Clear, legitimate deliverables spelled out in the contract
  • Fee is at fair market value 

1=Institution places no restrictions on any consulting and advising relationships.

9. Access of pharmaceutical sales representatives

Background: Industry sales representatives are employed to increase the sales of their company’s drugs. Permitting their access to medical staff is not in the interests of patients or staff.

3=Pharmaceutical sales representatives are not allowed access to any faculty or trainees in academic medical centers or affiliated clinical entities. However, faculty may invite other industry scientists who are not acting as sales representatives for specific discussions that do not involve marketing a specific product. 

2=Pharmaceutical representatives are allowed to meet with faculty and the following two criteria must be met: 

(a) meetings must take place only in non-patient care areas and

(b) meetings must take place by appointment only.

1=No policy, or a policy that does not substantially limit access.

10. Access of medical device representatives

Background:  Medical device representatives often provide faculty and staff necessary technical assistance and in-service training on devices and equipment.  It is inappropriate for them to actively market new equipment directly to physicians, particular while they are indirectly assisting with patient care.  Clarifying the non-marketing role of medical device representatives is a best practice.

3=Medical device representatives are permitted in patient care areas only for legitimate reasons not related to marketing, such as providing necessary technical assistance and/or training on devices and other equipment already purchased. 

(Exceptions to the requirement that equipment already be purchased can be made when faculty are involved in legitimate research under a signed contract to research new devices or new uses for approved devices). 

2= Medical device representatives are permitted in patient care areas and there is no specification or restriction regarding their activities. However, the policy does regulate site access in some way (such as requiring an appointment or registration).    

1= No policy, or a policy that does not substantially regulate access.

11. Conflict of interest disclosure

Background:  Disclosure is an important component of conflict-of-interest policies. Faculty should be transparent about any financial relationships that could potentially influence their clinical and educational duties.

3=Policy requires both of the following types of disclosure:

(a)   Internal disclosure to the institution, and

(b)  Disclosure to trainees/audiences

2=Policy requires at least one of the following:

(a)   Internal disclosure to the institution,

(b)  Disclosure to trainees/audiences 

1=No form of disclosure required, or no policy.

12. Existence of an adequate COI curriculum for medical students 

Background:  A formal curriculum on conflict of interest aims to teach medical students how to prevent marketing activities from inappropriately influencing their treatment decisions.

3=COI curriculum/education is required for medical students. The medical school’s curriculum materials that are submitted must reflect and cover most of the curricular content and objectives in the AMSA standards for a “model curriculum.”

2=COI curriculum/education is required for medical students but it is more limited.

1=No COI curriculum/education for medical students is required or there is no policy in place.  

13. Extension of COI policies to adjunct/courtesy faculty and affiliated hospitals/clinics

Background:  Policies written for medical schools and major teaching hospitals are generally understood to apply to faculty and trainees in those academic settings.  However, medical education extends beyond those walls to employees of all types and to community hospitals, clinics, and private practices.

3=Policy applies to both of the following:

  • All employees (full/part-time or volunteer faculty) and students/trainees
  • Applies wherever faculty and trainees are working (i.e.: affiliated institution, such as off-site clinics and offices, or community hospitals), even if the site does not have the same policy.

2=Policy applies to or is actively encouraged, for at least one of the following:

  • All employees (full/part-time or volunteer faculty) and students/trainees
  • Applies wherever faculty are working (i.e.: affiliated institution, such as off-site clinics and offices, or community hospitals), even if the site does not have the same policy.

1=Policy applies only to regular faculty and only within the primary medical center.  

14. Enforcement and Sanctions of Policies

Background:  To ensure compliance with policies, institutions must provide mechanisms for oversight and enforcement.

3=Policy states that there is a party (e.g. the committee or individual to whom violations should be reported) responsible for general oversight to ensure compliance with COI policies AND that there are sanctions for noncompliance (a description of sanctions is not required). 

2=Policy states that EITHER there is a party (e.g. the committee or individual to whom violations should be reported) responsible for general oversight to ensure compliance with COI policies OR that there are sanctions for noncompliance (a description of sanctions is not required).  

1=There is no policy for enforcement and sanctions of policies.

III. Determination of Letter Grade

Scores are computed by adding up the total scores for each of 14 domains. Since each domain can score from 1 (poor or no policy) to 3 (model policy), the possible scoring range for each school is 14-42.

Based on the raw score, a standardized score was calculated based on converting the maximum score, 42, to a more intuitive maximum score of 100. This formula is:

(Raw score X 2) + 16 = Standardized score.

This is then converted to a percentage by dividing by the maximum score and multiplying by 100.

Grades are assigned as follows:
A ≥ 85% (Excellent, or model COI policies; corresponding to raw scores ≥ 35)

B ≥ 72% (Moderate COI policies, raw scores 28-34)

C ≥ 56% (Poor COI policies, raw scores 20-27)

I ≤ 54% (Incomplete policies, raw scores ≤ 19)

Schools that have been newly founded receive a one-year grace period after they meet full accreditation (which occurs after the first class graduates). These schools are not listed on the Scorecard site.  If there are no policies submitted to us or no description of progress in developing such policies after a one-year grace period, such schools receive a grade of “I”.

Explanation of I (Incomplete)

In prior years, schools that did not respond to our requests for their policies were assigned an F. This year, approximately half of schools did not respond, a high number that may have been due to a number of factors, including resistance to the more stringent criteria for model policies. Rather than assigning all such schools Fs, we searched for their policies online, and in most cases were able to retrieve the necessary documents in order to assess their policies.

However, approximately 25 schools had very little online information available, preventing a meaningful assessment. In most of these cases, online material allowed assessment of three or fewer domains. Rather than assigning these schools Fs, we assigned them an “I” for “Incomplete

We will encourage “I” schools to furnish us with complete policies before the publication of the teaching hospital scorecard in September 2014. Schools that respond will be re-graded, and those that do not respond will be given a one-year grace period after which their grades will be changed from “I” to a “D”.  

Reliability of online policy searches

We used two different methodologies to retrieve policy information (assessing submitted policies submitted vs. finding policies via web-searches), and there is a possibility that the two methods are not equally effective at finding policies that actually exist. Some schools may have conflict-of-interest policies, but decide institutionally not to make them publicly available. If such a practice is common, it could call into question the validity of our web searching procedure, which could miss existing policies, and therefore lead to falsely low grades for such schools.

In order to assess the robustness of our search methodology, we randomly selected 15% of the schools which had submitted policies, and rescored them using web searches as our only methodology. In 9 out of 10 of these schools, the final grade did not change, indicating that our search methodology, while not perfect, is likely valid for the vast majority of schools. We will continue to encourage non-reporting schools to submit policies with a goal of ensuring that 100% of schools on the scorecard are graded based on the complete set of COI policies, whether available online or not.

IV. Reassessment

An institution may request an explanation or reassessment of its score within 30 days of the scorecard release if policies change or if it feels it has been scored inappropriately. The reassessment process requires that a school submit either (a) new or additional policies or (b) a written explanation of why the current assessment is incorrect, specifically addressing each relevant domain and referencing the policy and page number to provide evidence to the grading methodology. During reassessments, the grade will be changed to In Process and the reassessment completed within 60 days. Contact jmf@amsa.org for more details.

V. Assessment Instrument

The rating system discussed above was translated to a coding book to aid analysts in making their assessments. It can be downloaded as a pdf here.


Suggested Citation: AMSA PharmFree Scorecard 2014. <<Title of Web Page>> (online), <<URL of Specific Web Page>>, Sterling, VA: American Medical Student Association <<Date Accessed>>.



[i] The Pew Charitable Trusts, “Conflict-of-Interest Policies for Academic Medical Centers: Recommendations for Best Practices,” 2013, http://www.pewtrusts.org/en/research-and-analysis/reports/0001/01/01/conflictsofinterest-policies-for-academic-medical-centers.