Be Data Driven
No amount of data will lead to accelerated impact if it is not used to inform decision making. When an initiative is data driven, quality information is available to the right people when they need it, and they are using those data to take action. The data produced by a digital initiative should be used for more than just outputs, such as published work or donor reporting. Examples of the types of data that can be collected to inform decision making include surveillance, research, operations, project management and data from secondary sources collected outside of the program.
- Design programs so that impact can be measured continuously and incrementally, focusing on outcomes, not just outputs.
- Make use of existing data, including open data sets and data from interoperable systems.
- Use rigorous data collection methods. Consider and address potential biases and gaps in the data collected, perform data quality checks, and maintain strong documentation behind collected data.
- Close knowledge gaps by contributing data to the development community and using data and interoperability standards.
- Use quality real-time or timely data to support rapid decision making, improve programming for users and inform strategy.
- Present data in formats that are easy to interpret and act on, such as data visualizations.
- Create a data use culture by prioritizing capacity building and data use efforts across all stakeholder groups, including the groups whose data are being collected.
- Be holistic about data collection and analysis. Collect data from multiple sources, and use a mix of data collection and analysis methods. Analyze your data collaboratively with stakeholders.
- Identify and use open data and interoperability standards.
- Collect and use data responsibly according to international norms and standards.
Analyze & Plan
- Identify data needs in partnership with users and stakeholders. Users should drive the process of determining which data are needed for decision making, in what format and when collection should occur. Identifying the minimum data needed for decision making is also important as too much data can make it harder for users to distinguish what is important and make decisions quickly.
- Adopt standards to guide data use. Best practices and standards for data collection and management should be adhered to throughout the project lifecycle. For example, the United Nations published the following basic data principles in "A World That Counts: Mobilising the Data Revolution for Sustainable Development":
- Data quality and integrity
- Data disaggregation, ensuring that differences are understood by sex, age and other relevant characteristics
- Data timeliness
- Data transparency and openness
- Data usability and curation
- Data protection and privacy
- Data governance
- Data independence, ensuring collection of data by parties independent of the implementing organization
- Data resources and capacity
- Data (human) rights, including the right to be counted, the right to due process, the right to privacy and ownership of personal data, and a way to be removed.
- Identify available data sources that you can use. Draw from existing relevant open databases to minimize how much data your program will need to collect. These databases are license-free and include public data provided by government agencies, open source communities or other programs. Confirm that data are in a format you can work with and that they meet quality standards.
- Identify and address the risks associated with data privacy and security — especially around access to and use of personal data. Most programs will collect and manage sensitive information. Develop a plan to protect privacy, keep data secure and delete data that may no longer be necessary when the program concludes. Developing a responsible data framework can help to protect individuals from harm from data-driven activities, including discrimination from profiling or privacy risks.
- Assess data literacy and capacity for data production and use.
- Work with users and stakeholders to identify who in the ecosystem is using data and how they are using the data. Observe the role that data use plays in their daily activities.
- Identify skill gaps in data production or data use.
- Inventory the current data tools being used and identify where additional tools may be needed.
- Confirm that users can visualize data, quickly identifying what may be working or where improvements are needed. Get feedback from users on whether the data and visualizations are useful and if they have the data they need; if not, make changes to improve the visualizations.
- Develop a plan for filling data-related skill gaps, possibly through hiring or capacity-building activities.
Tips & Resources
- Tip: Look for milestones other than the midpoint and endpoint. Intentionally create milestones at points in time that make sense or align with other cycles, such as during quarterly or annual reporting or during school terms in education initiatives.
- Tip: Assess capacity and resources across data production and information use.
- Resource: Digital Impact Toolkit, Digital Civil Society Lab at the Stanford Center on Philanthropy and Civil Society.
- Resource: A World That Counts: Mobilising the Data Revolution for Sustainable Development: “Basic Data Principles,” United Nations.
- Resources: Examples of open data sources:
- DataBank: World Development Indicators, World Bank.
- Human Development Data Reports, UN Development Programme.
- World Health Statistics, World Health Organization.
- WorldPop, GeoData Institute at University of Southampton.
- Humanitarian Data Exchange (HDX).
- International Data & Economic Analysis (IDEA), USAID.
- Resource: 8 #globaldev Initiatives That Use Big Data Effectively, Devex.
- Resource: Open Government Data Toolkit, World Bank.
- Resource: Mapping and Comparing Responsible Data Approaches, GovLab at New York University Tandon School of Engineering.
- Resource: Data Use Partnership: The Journey to Better Data for Better Health in Tanzania, PATH.
Design & Develop
- Build an initiative that meets data accessibility and quality needs. Design the tool or system so that it provides information when it is needed for decision making. For example, real-time data can be important for operational activities, while timely data — such as data available on a monthly basis — may be needed for strategic purposes. You can define data quality minimums to set a level of “good enough” for decision making. Answer such questions as How reliable does the data need to be to make decisions with it? and When is timeliness more important than accuracy? Ensure that key data are continuously available to the initiative team and decision makers.
- Consider data transmission and storage needs. Develop automated processes for data extraction, preparation, analysis and reporting so that information is quickly usable, meaningful and available to decision makers.
- Build data literacy and a data use culture. Work with users to integrate data use into policies, practices and procedures. Data use can also be incorporated into job descriptions, training, performance reviews and incentive structures. Set up internal mechanisms to encourage regular data reviews and reflection. Identify data use champions who lead by example and can inspire others to use data for decision making. Engage users and stakeholders so that they see value in the data that they’re collecting — or the data that are being collected about them and understand how the data will be used.
Tips & Resources
- Tip: Focusing solely on collecting and storing data for later means data users don’t see benefits or value from the data. Ultimately, data quality suffers and the program has less impact. Help make data usable and useful for data users in real- or near-time so they become data champions.
- Resource: Data Demand and Use Concepts and Tools: A Training Tool Kit, MEASURE Evaluation.
- Resource: Big Data for Development: A Primer, UN Global Pulse.
Deploy & Implement
- Implement data privacy and security procedures. While doing this, continue to monitor for vulnerabilities in data collection and sharing, especially when new data sources or technological tools are being used. Evaluate how well your privacy protections are working. Validate that individuals have confidence that their data are not being used or shared improperly or with unauthorized parties, and address any concerns expressed. For example, individuals may worry that information will be shared with family members or other community members, or that the government could intercept it.
- Continue to build a data use culture. Focus on building overall data use skills and an understanding of the importance of data use. Consider collaborating with other practitioners to hold joint data use trainings. Identify and support individuals who can become data use champions, including those at the leadership level; they can encourage a data use culture and push for organizational changes needed to support the transition.
- Contribute to the community. Continue to participate in and contribute to the digital development community. Watch for new data sets, tools, case studies or other resources that would be helpful to your initiative. Help fill gaps in public data by appropriately sharing the data that you’re collecting so other community members can use and build on that data.
Tips & Resources
- Tip: Host capacity-building activities to build skills that support data quality and use.
- Resource: Open Data Handbook: How to Open up Data, Open Knowledge International.
- Resource: Fighting Ebola With Information: Learning From the Use of Data, Information, and Digital Technologies in the West Africa Ebola Outbreak Response, USAID.
- Resource: Big Data, Big Impact: New Possibilities for International Development, World Economic Forum.
- Develop and implement a monitoring and evaluation plan based on best practices for data collection and management and inclusive of all types of data requirements. Specify how data will be collected, analyzed, distributed and used. Plan for regular follow-up data reviews to measure impact incrementally. Set milestones that are based on when data are needed to make decisions about the initiative. Identify the data required to support an assessment of data use for decision making. The World Health Organization’s MAPS Toolkit includes a self-assessment that you can use to validate your evaluation plan.
- Collaboratively identify indicators and data to collect. Include measures on how the technology is being used and analytics generated by the technology itself. For example, are users always entering data on a specific day of the week? Are there trends indicating when users generate reports? Also, establish basic data quality standards and processes for review. For example, make it standard to include gender-sensitive indicators and sex-disaggregated data if women and men may be affected by the initiative differently.
- Develop a data analysis plan. Map out how the collected data will be organized, classified, compared and displayed relative to the evaluation questions, as well as how multiple data sources will be integrated. The plan should be written in conjunction with selecting data collection methods and instruments.
- Gather baseline data on your indicators, and regularly analyze data to measure impact and outcomes. Make use of existing data and data sets to assess the impact of your initiative, and also consider monitoring short-term outcomes. Review the data with users, stakeholders and the digital development community, as well as with funders, for additional insight. Consider how to share data usefully and meaningfully with partners; ask for partners’ help in interpreting the data and considering how to make improvements based on that analysis.
Tips & Resources
- Tip: Dashboards can help users easily monitor data quality and trends over time to identify where to focus data quality efforts and, ultimately, to improve programming. Structure dashboards so that high-level, key information is available to users at a glance.
- Resource: Data Demand and Information Use in the Health Sector: A Conceptual Framework, MEASURE Evaluation.
- Resource: Guidelines for Integrating Gender Into an M&E Framework and System Assessment, MEASURE Evaluation.
- Resource: New Data for Understanding the Human Condition: International Perspectives, Organisation for Economic Co-Operation and Development (OECD).
- Resource: Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment, World Health Organization.
- Resource: The MAPS Toolkit: “Self-Assessment Questions,” World Health Organization.
- Resource: Complexity-Aware Monitoring, USAID Learning Lab.