Author
InterAcademies Council
Enhancing the Capacity of African Science Academies
Research Contractor
InterAcademies Council
Release Date
September 30, 2015
Copyright
2014

Search this Publication

  • 1. Methodology of the Review

    The review described in this report was a mandatory part of the African Science Academy Development Initiative (ASADI) process, which called for both a midterm and a final review. It is intended to measure the extent to which the original objectives of ASADI had been met; to examine ASAD)’s further impact on academy organization, activity, and effectiveness; to describe the lessons learned during the ASADI process; and to "capture relevant insights regarding the methods employed by the ASADI program, sustainability, and future initiatives to maintain and expand built capacity" (see the complete statement of task in Appendix G). In the course of performing this last aspect of the task, the panel developed conclusions and recommendations about future capacity-building of African science academies. The contract to carry out this review was awarded to the InterAcademy Council (IAC).

    The midterm review was carried out by EnCompass LLC and OtherWISE: Research and Evaluation in August 2009, and completed in March 2010. It is an extensive and valuable document. But this final report is a free-standing review of ASADI and can be read without a prior reading of the midterm review.

    The final review that is the subject of this report was carried out by a panel organized by IAC. The panel was chaired by Professor Turner T. Isoun, formerly federal minister of science and technology of Nigeria, and was supported by a small secretariat. The review was carried out between October 2013 and November 2014.

    EVIDENCE GATHERING

    The panel used a range of techniques to assess the effects of ASADI, and to inform its views on the next steps for the academies themselves and for the systems of which they form part.

    Of these, perhaps the most informative was a series of site visits to the ASADI intensive- partner academies—the Cameroon Academy of Sciences (CAS), Ethiopian Academy of Sciences (EAS), Nigerian Academy of Science (NAS), Academy of Science of South Africa (ASSAf) and the Uganda National Academy of Sciences (UNAS). These took place between March and May 2014.

    Each visit involved at least one panel and one secretariat member in structured discussions with academy staff at all levels, especially those who had been involved in ASADI-led training, and with academy members and leaders. In addition, interviews were held with a wide range of individuals in government, higher education, and other stakeholder organizations that had interacted with the academies. The host academies identified the stakeholders that were interviewed during the site visits. The panel identified additional experts that were interviewed by panel members and staff during the site visits, or by telephone or Skype. The panel is grateful for the generous amounts of time made available for these interviews, often by busy people with significant responsibilities. Their willingness to participate is itself testimony to the importance of ASADI. They spoke to the panel on a confidential basis and are not quoted directly in this report except by specific consent. A total of 102 individuals were interviewed during the site visits—57 were academy stakeholders (leadership, members, and staff) and 45 were external stakeholders (from government, nongovernmental organizations, academia, and industry). Apart from the site visits, an additional eight experts were interviewed by panel members and staff, mostly by telephone or Skype.

    Two members of the ASADI review team, Panel Chair Isoun and Study Coordinator Dorothy Ngila, provided continuity by attending all of these visits, which were crucial to the review’s understanding of the issues facing the very diverse group of ASAD) partner academies.

    A second major input to the review was a pair of questionnaires completed by the academies. One focused on the key experiences of the academies during ASADI. The other was designed to help the panel evaluate how well the ASADI program objectives had been met. They examined issues such as the academy’s role in its own national science system, the changes in its standing resulting from ASADI, its role in supporting other national academies, and continuing challenges which might be addressed in coming years.

    The third standardized input for the review was a grid-style document termed the Evolution Table. Each academy prepared this table itself. It was used to capture changes in the academy’s status in every field from its legal standing to its policy influence, its internal and external communications, and its organizational and financial capacity, with particular reference to ASAD)’s contribution to its recent development. These make up Appendix I of this report.

    The U.S. National Academy of Sciences (USNAS) ASADI program board and staff also provided extensive input to the review. The chair and director of ASADI spent several hours with the panel at its January 2014 meeting in London, and other ASADI staff members joined the discussion by videoconference. Questionnaires were completed by ASADI staff members, including the director and the country directors that worked closely with CAS, EAS, NAS, ASSAf, and UNAS. The former country director that worked with ASSAf was also interviewed. The ASADI staff also held teleconferences with panel members and staff prior to site visits. This allowed the panel to examine the level of agreement and disagreement between ASADI participants in Washington and Africa on the reality of the ASADI experience. A comparative analysis of the answers was conducted as part of the data analysis described below.

    USNAS also provided an extensive amount of written material concerning ASADI. As the catalogue of inputs to this review (Appendix C) makes clear, the ASADI project has been richly documented. As well as material generated specifically for the review, the panel was able to consult the original project material submitted by the USNAS to the Bill & Melinda Gates Foundation; ASAD)’s own annual reports; the midterm review; a lengthy list of publications from each of the academies, and copies of many of these publications, including material with both a policy and a scholarly orientation; a range of policy, organizational, and financial documentation about each academy; interviews with the USNAS ASADI team members and written contributions from them; and a wealth of ASADI training documentation, itself a key output of the project. The catalogue of academy publications comprises Appendix D.

    The Council for Scientific and Industrial Research (CSIR) was commissioned to carry out the data analysis component of this evaluation. The data analysis used qualitative and quantitative methods to:

    1. Collate, synthesize, and summarize annual reports and other documents that provide information on the ASADI project with the aim of identifying key themes emerging from the reports;
    2. Summarize financial statements and information over the grant period;
    3. Produce a summary of the outputs by academies in the form of workshops, conferences, and publications;
    4. Catalogue all the information received so far for the evaluation; and
    5. Identify data gaps in the documents analyzed.

    At a Washington summit meeting in August 2014, ASADI participants were asked to carry out an informal SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis of the ASADI experience. An edited summary of their responses comprises Appendix H, and provides further evidence to support the findings of the Review Panel.

    In addition, material was collected on ways in which the academies collaborate with each other and with other academies, for example via AMASA, the Annual Meeting of African Science Academies.

    The majority of the evidence available to this review concerned the five primary ASADI partner academies, also referred to as the "intensive partners." This report does not deal in detail with the smaller-scale ASADI activities in Ghana, Kenya, and Senegal, known as the ASADI secondary partners, nor with ASAD)’s involvement with the African Academy of Sciences.

    The volume of material made it obvious to the panel that a further analysis phase would assist the writing of this report. This phase of the project involved commissioning experts from South Africa’s CSIR to synthesize and summarize the documents and outputs, identifying specified themes and summarizing ASAD)’s financial statements.

    The next stage of information-gathering was a round of telephone and Skype interviews, intended to fill gaps in the panel’s knowledge or to reach people whom it had proved impossible to meet in person. Finally, there was a later round of questionnaires which the academies completed on specific topics of interest such as the use of forums to develop policy advice.

    STRUCTURE AND DEVELOPMENT OF THE REPORT

    The overall direction and structure of this report were the subject of two panel meetings, the first hosted by the Royal Society of London in January 2014, and the second by ASSAf in Pretoria in May 2014. Following the second panel meeting, a draft report was prepared, which was then finalized by the panel through a series of conference calls and other exchanges. The panel’s draft was then reviewed by external experts according to )AC’s standard procedures.

    It is notable that when asked in the questionnaires about their progress under ASADI, the academies were more modest about their achievements than were external observers such as government. This modesty was one principal difficulty for the review process itself.

    Another complicating factor is that even without ASADI, these academies would not have stood still during the 10-year period under review. So it would be wrong to attribute all positive change in the academies to ASADI. They would have changed of their own accord, and via the support which many of them received from academies in the developed world and their governments.

    It is important to stress that this report is a review of ASADI. It is not a review of African science academies, or of any other organization or network. And it is certainly not a review, assessment, or appraisal of any individual.

    VALUE OF THE REVIEW EFFORT

    The IAC ASADI review was a considerable undertaking, appropriate to an exercise of the cost, scope, and duration of ASADI itself. It involved a wide range of information-gathering from many individuals and organizations with direct or indirect connections to ASADI and the academies it supported. The panel believes that this level of review adds value beyond confirming that ASADI met its nominal objectives, by highlighting a broader range of ASADI outcomes and providing a focus on the future development of African science academies.