Cosmo Howard (University of Victoria, Canada)

Modern nation-states rely extensively on data to govern the complex social, economic and political systems that operate within their borders. Assembling data that is relevant, timely and of sufficient quality to aid official decisions has always been important, but in the age of "evidence-based decision making" it is regarded as an essential prerequisite of effective governance. To help meet policy makers' informational needs, governments invest considerable resources into national statistical agencies, which are tasked with producing and disseminating analyses of economic conditions, social issues and policy problems. In the past, these agencies enjoyed a virtual monopoly over the production of official statistics, but in recent decades their privileged position has been progressively eroded as new producers of government statistics have emerged in other parts of the state, including central agencies and policy ministries. Many of these new sources of official statistics are closely connected to policy and political processes and do not enjoy the arm's length independence traditionally afforded to statistical agencies. As a result, some critics (Hoffmann 1995; Holt 2007) argue that government statistics have achieved greater policy relevance at the expense of political independence, methodological rigour and public trust. 

In light of these developments, my project on Governance Challenges for National Statistical Agencies asks: how should we understand the relationships between politics, policy and knowledge in the context of the dispersal of statistical production to multiple organizations within state? This paper provides a systematic survey and synthesis of current understandings of these relationships. It draws upon two important strands of the contemporary governance literature. The first is the literature on post-bureaucractic administration and horizontal management, which is focused on the shift from hierarchical to networked relationships and offers a set of concepts and theories for exploring how statistical agencies are orientating themselves towards the policy process and to new rival producers of official statistics. The second "governmentality" literature explores how states have come to rely on statistics to govern subject populations, and makes the point that official statistics are not neutral or static, but shaped by prevailing policy discourses and political ideologies. A key innovation of this paper is to connect these two separate strands of the governance literature, and to suggest how key concepts such as "relevance", "rigour" and "independence" must be rethought in the context of distributed statistical governance.

Cosmo Howard (University of Victoria, Canada - howardc@uvic.ca)

David Pontille (CNRS, EHESS), Didier Torny (INRA)

In a growing number of countries, public administrations and agencies seek to systematically evaluate the scientific outputs of their universities and research institutions. The impact factor (IF), fashioned by the Institute for Scientific Information, was for a long time the gold standard for the ranking of scientific productions according to their forms of publication. Nevertheless, even when used, IF was less accepted and integrated into social sciences and humanities practices. In this paper, we will focus on the journals ratings produced by the European Science Foundation (ESF), the French Agency for Evaluation Research and Higher Education (AERES), and the Australian Research Council (ARC). In these three cases, the challenge is to shape alternative policy tools to the IF, which clearly favours the English-language productions, to evaluate and promote research in social sciences in a multilingual context. The manufacture of these rankings is based on the mobilization of experts and institutions to assess the quality of journals and rate them (A, B, C,).

Points of views on these journals ratings are heavily adversarial. Some advocate for a realistic account of scientific hierarchies, based on knowledge produced by the most relevant epistemic communities: ratings are objectified representations commonly shared, of inherently limited use, whose possible effects are sought. Others judge ratings as performative and based on criteria controlled by research management institutions. Thus, they are considered as policy tools which produce managerial standardized assessments and whose uses are dangerously manifold. To go beyond this opposition, we have analyzed the numerous texts produced by the three research evaluation agencies and the diverse public statements (editorials, petitions, points of view, scientific articles...) on ratings.

We will first stress the wide range of ratings processes, going from a fully open advisory process to the gathering of a few anonymous experts. Then, we will demonstrate that journals ratings draw on two different kinds of criteria, whether explicitly foreseen or published afterwards. On the one hand, experts take into account quantitative data on journals audience: board lineup, the number and national origins of submitting authors, journal broadcasting (number of subscriptions, downloading...). On the other hand, they strictly rely on procedural norms: reviewing process (single or double blind), unsolicited articles, number of referees, publication delay. Our analysis will also underline the role of journals themselves, which took part in the ratings controversies. Their criticisms resulted in a substantive revision of journals lists (ESF and AERES) and the production of a particular compromise around a 'scientific perimeter'. Henceforth, such a perimeter defines what sound academic knowledge is and reconfigures the politics of evaluation devoted to individuals, laboratories, universities or countries. These new ratings are gradually seen as coproduced by legitimate fractions of disciplinary communities. Therefore, they might have more impact on social sciences and humanities research than former bibliometric tools. 

David Pontille (CNRS, pontille@ehess.fr), Didier Torny (INRA, torny@ivry.inra.fr)

Luís Miguel Carvalho, Natércio Afonso , Estela Costa, Institute of Education, University of Lisbon

This presentation is focused on the growing use of regulatory instruments which entail the production and dissemination of knowledge, studying their production and diffusion and their use by the decision-makers for whom they are intended.

Our presentation brings together results of several studies on the fabrication, circulation and use of a policy-knowledge tool: the Programme for International Student Assessment (PISA) produced by OECD.  These inquiries focused on the supranational construction of PISA and the different social and cognitive trajectories of the instrument in 6 countries: Belgium, France, Hungary, Portugal, Romania, and United Kingdom-Scotland.

The comparative analysis provided the opportunity to characterize the survey as a very powerful policy instrument whose characteristics allow for its successful penetration into very different policy-making regimes and very diverse political circumstances. These characteristics were identify as its credibility (resulting from its scientific and technical sophistication, but also from OECD reputation as a trustworthy "truth-teller"), its adaptability (because of its multipurpose nature allowing for adaptation to different political contexts, providing rationales and justifications for diverse and even opposite policies, therefore used as an "empty container" to support and to oppose the very same policy measures), and its pertinence towards a common set of policy trends (the raising awareness rhetoric, the modernization narrative, the discursive shift to comparison in the educational policy debate, the "naming and shaming" politics, and the cross-national concern about the relevance of context variables in the students achievement results).

The PISA tool is developed around conventions about what is "knowledge for policy", how it should be produced and what is the credible knowledge to mobilize into the survey framework. Thus, it considers the knowledge produced within the PISA world, how it is used by policy actors, the problems and solutions raised or endorsed concerning the educational systems and their appropriate governing, and which are activated in national public action. Due to this production processes, the PISA knowledge shows a significant level of flexibility, as it becomes more and more close to the audiences moving from "revelation" (the core results of the survey) to "explanation" (the interpretation of the results through relationships between variables), and to "condensation" (findings conceived as policy questions).

The PISA tool is defined as a multipurpose tool, a resource to legitimate the actions of those who refer to its results and its explanations. Structured around the concept of regulation, the study addresses the political and cultural specificities and the variability of PISA national receptions, but also the «attractiveness» of PISA as a space for knowledge creation and exchange and an "unavoidable" source of information for policy-making.

Defining PISA as a KRT means that it is conceived as an example of the complex and circular relationship between knowledge and policy: PISA as a policy instrument produces knowledge; PISA as a research instrument produces policy.

Luís Miguel Carvalho (lmcarvalho@ie.ul.pt), Natércio Afonso (natafonso@yahoo.com ), Estela Costa (ecosta@fpce.ul.pt ) Institute of Education, University of Lisbon

Eszter Berényi (Eötvös Loránd University, Faculty of Social Sciences, Budapest

The proposed paper discusses issues related to the fabrication of the first national student assessment system in Hungary, the "Assessment of Basic  Competences" ("ABC").  Policies on educational measurement are among the most knowledge intensive subfields of educational policymaking; and vice versa, educational measurement is the most "policy-sensitive" of all sub-fields of educational science. The rise of measurement regimes and the strengthening of the accountability approach is an important landmark on the visionary road towards the ideal of evidence based policy making accompanied by the politization of science. The ABC signs a clear shift in the knowledge/policy relationship in the Hungarian educational policy making: it presents several examples of how the boundaries between knowledge and policy are blurred in the process of postbureaucratic governance. When analysing the story of the ABC we will examine how the political and the knowledge aspects of this educational measurement mutually shaped each other; how coalitions and compromises were made between the main actors from the knowledge and policy fields; and how all these developments contributed to the emergence of special artefacts and hybrid actors. Several issues and concepts related to the ABC, such as  the so called "pedagogical added value" or the sample size  will be treated  here as belonging both to the  fields  of knowledge and  policy. Their importance cannot be understood without taking into consideration both of these fields as their meaning is constructed in both of these spheres at the same time. Our hypothesis is that the configuration of these concepts is what makes them so influential in the process of policy construction. The analysis of the story of the ABC draws heavily from the concept of "bricolage" (Ball 1998). Bricolage is a key concept for the interpretation of the different layers of meaning of the ABC, and it helps us understand how diverging expectations toward the ABC  as a knowledge based regulation tool can very well coexist, shape each other and eventually become aligned. 

Eszter Berényi (Eötvös Loránd University, Faculty of Social Sciences, Budapest, beresz@webstation.hu)  

Justin O. Parkhurst (Health Policy Unit, London School of Hygiene and Tropical Medicine

It is well known that ideology influences political decisions, and may affect the selection of evidence in political argument. However, deeply held ideological and moral positions may equally shape the understanding of evidence itself, affecting the seemingly 'rational' construction of knowledge. This will have direct implications for the conclusions policy actors draw on how a given body of evidence leads to policy recommendations. This phenomenon is explored through an analysis of the HIV prevention policies constructed as part of the United States' 'President's Emergency Plan for AIDS Relief' (PEPFAR). Since 2003, PEPFAR has been one of the largest supporters of HIV/AIDS activities in low income countries. However, under the George W. Bush administration, it faced a high level of criticism over its HIV prevention strategy known as ABC (said to stand for Abstain, Be faithful, or use Condoms). Numerous critics claimed that PEPFAR's interpretation of ABC was ideologically driven, and undermined HIV prevention internationally. Supporters and official PEPFAR documentation argued the approach to be 'evidence-based', and derived from data on Uganda's successful HIV experience. Those critical of PEPFAR have drawn their own conclusions about the policy implications of Uganda's historical HIV experience and the meaning of ABC. This paper analyses discourse from both supporters and opponents of the Bush administration's HIV prevention strategies to investigate the understanding and uptake of evidence from a critical political perspective. It applies cognitive framing concepts to explore how underlying belief systems can lead different interpretations of evidence, leading to subsequent divergent policy arguments.

The study finds that two competing moral belief systems can explain many of the differences over what ABC is said to be, and how evidence is understood in practice. Actors believing in a single morally-correct way to behave tend to embrace understandings of ABC which emphasise abstinence for youth and fidelity in marriage. They are more critical of condom policies and legitimise their arguments with understandings of evidence which are internally consistent with their belief systems. In contrast, those possessing a core belief in freedom of lifestyle-choice demonstrate contrasting understandings of ABC, arguing it should be a comprehensive set of choices individuals can select from. Condom promotion is much more acceptable to this group, while promotion of abstinence or fidelity are problematic (dictating how to behave). Again, the understanding of evidence (such as the burden of proof of effectiveness of different interventions, or the lessons drawn from Uganda) appears to be linked to how internally consistent that evidence is with the underlying belief system.

Using such a cognitive-political framework can help to understand the political considerations behind the use of evidence in many controversial policy debates, illustrating that ideology not only influences preferences on policy outcomes, but also shapes the understanding of scientific evidence itself.

Justin O. Parkhurst (Health Policy Unit, London School of Hygiene and Tropical Medicine, Justin.parkhurst@lshtm.ac.uk)

Bori Fernezelyi, Julia Koltai, Sara Levendel

Knowledge-based regulatory instruments are typical and increasingly widespread forms where knowledge and power merge in the policy field. We define knowledge-based regulatory instrument as a mechanism, object, tool, or process specifically concerned with diffusing a particular kind of knowledge in order to shape the behaviour of actors in a given policy domain. In this paper, we endeavour to demonstrate the functioning of such an instrument through the analysis of the genealogy of the DRG system in the Hungarian health sector. Furthermore, we investigate the different interests, values, and views of the groups and individuals involved in the conception and promotion of the instrument.

In Hungary the financing of the hospital sector is based on the performance-based DRG system (disease related groups), which aims to rationalize the allocation of treatment costs and takes into consideration both medical and economic rationales. The rise of the DRG system is a typical case of the parallel evolution and later fusion of an expert knowledge system and political demand; From the 1970's the development of the computer sciences was adopted to facilitate administrative needs of the medical field and soon it became an evaluating tool. From the 1990's onwards political demand was rising for a more rational financing of the health sector. After the system change a health reform was needed and the most elaborate alternative for that was the Hungarian version of the DRG-system finally introduced in 1993. Though, the model originates from the United States it's not explicitly „best practice" as it was not adopted, but developed nearly independently. It is one of the first, and believed to be one of the most influential, among the health policy reforms after the political change in 1989.

Essentially DRG is a financing regulatory tool, thus it leans on economic rationales, but at the same time it is based on medical knowledge such as medical protocols and guidelines. This knowledge-based regulatory instrument that combines the knowledge of evidence based medicine and the economic aspects, induces self-regulation and optimal functioning in the in-patient service. 

Bori Fernezelyi (febori@gmail.com), Julia Koltai (Koltai.Juli@socio.mta.hu), Sara Levendel

(Levendel.Sara@socio.mta.hu)

Jennifer Smith-Merry (University of Edinburgh)

This paper will explore the relationship between the World Health Organization's Regional Office for Europe (WHO Europe) and Scotland in the context of mental health. It will examine why this relationship is fostered as an important one by both actors and the instruments through which they legitimise the relationship.

The paper begins by discussing the historical relationship between Scotland and WHO and the official relationships which regulate their interactions. It will then examine in detail Scottish involvement in WHO Europe's Ministerial Conference on Mental Health in Helsinki in January 2005 which resulted in the signing of the Mental Health Declaration for Europe and its accompanying Action Plan in order to determine how and why the relationship between the two actors is built.  The analysis draws on a series of interviews with key actors involved in the development of the Declaration and Action Plan alongside an analysis of related texts. Our main finding is that a mutually validating relationship between the two actors is created and perpetuated through personal communication, meetings and the joint production of documents. 

Jennifer Smith-Merry (Politics and International Relations, School of Social and Political Science, University of Edinburgh, j.l.smith@ed.ac.uk )

Cerfontaine Gaëtan (FNRS -University of Liège)

This presentation proposal rests on empirical data collected through a European program named « Know and Pol ». The aim of our paper is to understand the process through which the recommendations emitted by the World Health Organization are translated in on-the-ground policy and practices in the Belgian mental health sector.

As for many countries, these recommendations and the care model they promote contrast on several points with the practices and the structure of the Belgian mental health sector. One element advocated by the WHO has in particular raised controversies and oppositions: the necessity to replace the traditional hospital-centered model by community cares within small ambulatory structures. Therein lies the principal criticism addressed to the Belgian mental health cares policy.

Through a focus on the reception of these recommendations and on a particular policy - "the therapeutic projects in mental health", which can be seen as the last tentative to reform the organization of the sector -, we will analyze a process through which "big ideas" are translated in concrete policies.

Indeed, in spite of the lack of legal constraint, these recommendations have proved to have some kind of normative capacity. Nowadays in Belgium, every text or political discourse (held by politicians, civil servants or representatives of the field actors) concerning the mental health policy acknowledges the WHO's recommendations and this, despite the fact that they could be seen as antagonist to some actors' interests. In order to understand this evolution, we shall refer to the Michel Callon's concept of "point of obligatory passage" which underlines the creative activity of the actors who do not just receive passively these norms but, on the contrary, who reinterpret and translate these ideas in new policy. Our presentation will underline (1) that in spite of the antagonism between those ideas and some actors' constituted interests, notably the representatives of the hospital organizations, these actors may reinterpret WHO norms strategically so as to fit with their own interests and the organizational context they are involved in. (2) We assume that such an interpretive work is possible because these recommendations do not provide a clear way to act but, can be used in order to imagine "new possible worlds". (3) By doing so, these actors are legitimizing WHO's recommendations and are reinforcing the possibility for other actors to use them in order to elaborate and legitimize new policies.

In conclusion, we'll see that the process of reform and the influence of these norms are far from being direct and linear but, on the contrary, are long, iterative and constituted of "reinforcing loops". At the methodological level, we'll discuss the possibility to observe the multi-level process of elaboration of a policy through the materials (texts, declarations, evaluations...) produced and mobilized by the different actors. Indeed, with the interviews and observations, these materials constitute one of the major parts of our data collection.

Cerfontaine Gaëtan (FNRS -University of Liège, G.Cerfontaine@ulg.ac.be)