Evaluating Evidence and Information Sources

In this workshop, participants will learn how to analyse and verify collected information, as well as how to evaluate information sources to be able to assess the reliability of research / investigation findings.

Learning Activities

Opening (15 minutes)

Instructions for trainer

• Grab attention by posing a question or commenting on a relevant picture.
• Introduce yourself and the goals of the workshop.
• [Optional] Introduce the source of the workshop material (Tactical Tech)
• Introduce the workshop agenda.
• Suggest ground rules for the workshop. Ask participants whether they would like to modify your suggestions or suggest other rules. Ensure that everybody understands and agrees with the ground rules. Specific suggestions about setting ground rules are available in the Facilitator's Guide, section on "Delivering the Workshops".
• Establish the dynamics for group work, explain that participants will work in the same teams during the interactive workshop activities.

Instructions

• Make a quick round of introductions by asking participants to share the following. You can choose 1-3 of the following questions:

  • Where are you calling from? / Where are you from?
  • What is your work about?

• Alternatively, you can pick an icebreaker exercise that encourages participants to get creative by drawing answers or ideas on an online whiteboard or, if off-line, stand up and perform some tasks or discuss in groups. Check the Icebreakers section in the ETI Facilitator's Guide for inspiration.

Introduction (15 minutes)

Tools/ Materials

• Shared cloud folder and files / text edit platform (e.g. Framapad) for listing observations and voting (if online.)
• Individual sheets of paper, sticky notes / post-its, pens (if offline.)
• "Evaluating Evidence – Activity Templates", section "Individual Activity: What do you see?" to download / print or fill out online; alternatively, a shared digital whiteboard or pad with the template (such as Miro, Mural, etc.)

Instructions

[5 minutes]

• Ask participants to individually think of an observation from their context that they find interesting or story-worthy:

  • Something that they find suspicious, alarming, or surprising.
  • It could be a phenomena that's affecting the community as a whole or a very local issue.

• Participants write their observations on a shared document with an empty table for entries, in maximum three lines.

[10 minutes]

• All participants are asked to vote for the observation that they'd like to investigate by marking an X next to it. Note that:

  • Each person has two votes to use.
  • They can vote for their own observation if they want.

• After allowing a few minutes for votes, count results and choose the observation that gets that most votes.

  • If more topics receive the same score, you can pick the one that seems more complex to allow for various hypotheses).
  • This will be the investigation topic that all the groups will develop during the workshop.

Developing a hypothesis (25 minutes)

Instructions

• Explain that the goal now is to know the story behind this observation, why this is happening.

• Ask participants:

  • " What do you think is the best way to approach this?"

• Hear out a few thoughts about this.

• Depending on the suggestions, you may notice that most answers are based on a hypothesis , so you can point that out.

• Or you can simply say that the focus in this workshop will be on working with "working" hypotheses and explain why – because a hypothesis helps us plan and keep track of the sources and information we need to gather.

• Briefly explain what a working hypothesis is: "a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation" (definition source: lexico.com).

• Highlight that:

  • Developing a hypothesis gives you something to verify.
  • This helps you structure and plan your investigation.
  • It is important to recognise that your starting hypothesis is not set in stone (i.e. it can be modified and adapted along the way, as you conduct the research).

RESOURCE:

• Guide (main source for this workshop): "Evaluating Evidence and Information Sources"/ section "Developing a Hypothesis", "Exposing the Invisible: The Kit", Tactical Tech.

Tools / Materials

• "Evaluating Evidence – Activity Templates", section "Group Activity: Develop a hypothesis"
• same as before.

Instructions

[15 minutes]

• Divide participants into small groups.

• Ask each group to:

  • List as many hypotheses as they can for the observation from the first exercise. (You can encourage them to go wild.)
  • By the end of the small group work time, each group should choose one hypothesis that they think is most likely.

[5 minutes] Debriefing

• Participants go the main room and share their finding.

• Each group shares, in 1 minute, what kind of hypotheses they came up with and, which one they chose at the end.

• Ask participants to mention if they found it difficult to agree on a hypothesis and what determined them to make that choice at the end.

Finding evidence (60 minutes)

Instructions

• Ask participants:

  • "What do we do with this hypothesis now?"

• You will probably get answers with the "keywords":

  • verifying,
  • proof,
  • evidence, etc.

• Ask participants "Why do we need evidence?"

• Allow a few minutes for them to share thoughts.

Debriefing

• Highlight that:

  • Investigations are only as strong as the evidence you collect for them.
  • In some cases, you may even face legal consequences if you are not able to support your assertions with strong evidence.

Tools / Materials:

• Slides for a power point presentation or pre-written flip-chart papers (for offline)

Instructions

• Explain that the goal now is to collect enough evidence to get as close as possible to proving or disproving the hypothesis.

• Prepare and make a short presentation covering the following points:

  • What evidence is": a fact or piece of information that shows that something exists or is true."
  • Your goal is not to prove that your hypothesis is correct at all costs.
  • Instead, your responsibility it to keep an open mind about alternative explanations, even if they do not agree with your own beliefs or convictions. This shows that you can be an impartial and thus reliable investigator / researcher.
  • Keep in mind that it is extremely rare for any investigation to provide 100% proof.
  • There will be situations where – despite your best efforts – your investigation will reach a dead end.

Tools / Materials

• "Evaluating Evidence – Activity Templates", section "Find Evidence & Classify evidence", only column 1.
• Same as before.

Instructions

[15 minutes]

• Ask participants to return to their groups.

• Ask each group to brainstorm and make a list of possible pieces of evidence related to the issue and hypothesis they chose in the previous activity.

• Tell participants that the pieces of evidence don't have to confirm their hypothesis, they can also end up contradicting it.

• Mention that their list will then be shared with another group to work with.

• Give them ideas of what can be included in the list, for example:

  • social media posts,
  • official documents,
  • an old interview,
  • a phone call,
  • a report,
  • … etc.

• Ask them to be specific, e.g.:

  • not to list "a report" but rather "a report that shows the number of women who drop out of a certain university course."

[10 minutes] Debriefing

• Participants go the main room and share their finding.

• Each group shares, in 1-2 minutes, what evidence types they came up with.

• Ask participants to mention any challenges they see with this evidence and whether they expect that some of it may contradict their hypothesis.

Tools / Materials:

• Slides for a Power Point presentation or pre-written flip-chart papers (for in-person).

Instructions

Explain that evidence can be classified according to several categories:

• Type:

  • Direct evidence: speaks straight to your hypothesis and establishes facts (e.g. video, audio and photographic evidence, eyewitness accounts of events, official documents)
  • Indirect evidence: supports an assertion but is not strong enough and mostly second-hand (e.g. statements by people who did not observe events directly, statements by spokespersons for others)

• Source:

  • human – obtained from people
  • physical – e.g. water samples, court documents, books, video/photo (raw format from the camera)
  • digital – e.g. information that is found on the internet, e.g. online articles / studies, digital databases, online videos / photos, blogs, social media, discussion forums

• Origin:

  • desktop research
  • field research

• Note that with some effort nearly everything in the digital world can be manipulated, including audio, images and video.

  • The metadata of photographs that you have not taken yourself can be very easily altered and should not be relied on as evidence, unless you can verify and confirm it from its original source(s) and/or by other means such as reverse image searches.

RESOURCE:

• When preparing this presentation, use content and examples from the "Evaluating Evidence and Information Sources" guide, section on "Types of evidence and searching for proof", Exposing the Invisible: The Kit.

Tools / Materials

• "Evaluating Evidence – Activity Templates", section "Find evidence & Classify evidence", remaining columns.
• Same as before.

Instructions

[10 minutes]

• Ask participants to return to their groups (same as before) but shuffle the lists of evidence, so that each group works with another group's list. — From now on, they will work on these new lists until the end of the workshop.

• Each group categorizes previous evidence according to the types above, using the templates provided.

[10 minutes] Debriefing

• In the plenary room, participants share their classification, and what they found difficult to classify and why.

Analysing evidence (50 minutes)

Tools / Materials

• "Evaluating Evidence – Activity Templates", section "Evaluate evidence".
• Same as before.

Instructions

[10 minutes]

• Participants return to their groups and continue working based on the shuffled lists they had in the previous exercise.

• Ask each group to evaluate the pieces of evidence they have as:

  • Weak
  • Medium
  • Strong

[10 minutes] Debriefing

• Go back to the plenary room and share findings.

• Ask participants how they evaluated the evidence and what makes a weak or a strong evidence for them.

• Ask them how the strength of the evidence can change by changing the context.

Tools / Materials

• "Evaluating Evidence: Bias Cards" (Word file) — Note: you will need to download and prepare these eight cards: four with "cognitive biases" and four with their "related examples." You can use the template to create online cards on a Miro / Mural board or print them out and cut them to be mixed on a board/table, if conducting the workshop in person.

Instructions

[10 minutes]

• Briefly explain what a cognitive bias is:

  • "A systematic error in thinking that occurs when people are processing and interpreting information in the world around them and affects the decisions and judgements that they make."

• Divide the participants into small groups – this time they can be random, no need to stick to previous groups.

• Mix the eight cards (provided here) in random order and ask groups to match the biases with the related examples.

• They can quickly look up the biases.

Debriefing

[5 minutes]

• Provide the correct answers and ask participants if there are other biases that they experienced first hand or observed.
• They don't need to give the formal name of the bias, just to explain what happened.

Instructions

• Lead a short discussion asking participants:

  • "How can one mitigate the risk of bias?"

• At the end of the discussion, you may highlight the following if not mentioned:

  • Seeking second opinions of evidence interpretation by colleagues and other investigators
  • Reviewing your conclusion after some time has passed
  • Making an effort to disprove conclusions and evidence interpretation
  • a rule that is often applied in journalism requires at least three sources that are independent from each other to confirm the same information before it can be published.

• Move on to asking:

  • "What are other risks to your information/evidence i.e. factors that reduce their validity?"
  • Give as example: outdated information

• Conclude by asking:

  • "What are some mitigation actions to be taken to deal with these risks?"

Managing evidence (15 minutes)

Tools / Materials

• Slides for a Power Point presentation or pre-written flip-chart papers (for offline)

Instructions

Prepare and give a short presentation covering the following points:

• Storing evidence

  • For digital evidence, it is important to keep digital copies on separate hard drives, which should be stored in different spaces to your computer.

  • Internal and the external hard drives should be fully encrypted.

    • If you live in an area where you face significant personal or legal risk from your investigative activities, keep some copies in a safe place outside of your own home.
    • Keep in mind that you may be putting other people at risk by storing evidence with them.

  • Keep a physical copy or any original documents such as letters but also scan / photograph physical documents to create digital copies.

    • Keep these encrypted on your internal and external hard drives.

  • Environmental samples can degrade quickly (especially water samples) and they should be processed by laboratories right away rather than kept in your home / office.

  • Complex and long investigations can result in vast amounts of evidence gathered and in some cases you might not understand the full relevance of evidence until later in your investigation.

    • For example, an image of the polluted river with GPS coordinates could become more relevant after you receive the water testing report from a lab, which points to a specific location.
    • A good way to manage your evidence is by creating an evidence sheet that allows you to quickly locate evidence in your collection.

  • Show an example of an evidence sheet (you can use the example from the "Evaluating Evidence" guide, section: "Managing Evidence"

• Protecting evidence

  • There are several resources you can use to learn about how to better assess your context and risks, take actions such as encrypting your hard drives and protecting your communications, in particular with confidential sources.
  • Recommend resources to start with such as these guides and articles: "Safety First!", "Risk Assessment Is a Mindset, Not a Checklist", "Security in a Box" tutorials, etc.

• Sharing and publishing evidence

  • Some reasons why you might consider publishing the evidence you have collected, processed and verified include:

    • It is in the spirit of transparency, especially when considering that many investigations focus on uncovering what is unjustifiably hidden from us.
    • It allows for review and scrutiny of the conclusions you reach and can be part of a necessary review process.
    • It can result in additional information being unearthed by people who may be motivated and inspired by your investigation. Others might have access to more data or they might be interested in investigating the issue themselves.

  • There are different levels of information-sharing such as sharing with co-investigators, people in your network, activists, NGOs, journalists, or providing general public access to your evidence.

    • As a rule, the identity of confidential sources should never be revealed and neither should any information be published that provides clues about who they are.
    • Sometimes you might end up purchasing data from commercial providers such as corporate registries, trade registries or land records. In some cases, such providers will prohibit you from republishing the data and information you buy from them. - While it is a good idea to avoid legal risks by not republishing data that was purchased when such restrictions exist, it is normally not a problem to refer to and reference key quotes of that material in the public interest. Commonly this falls under the "fair use" rule for copyrighted materials.

  • When you publish information , consider how do it in accessible formats so that others can easily understand and use it.

  • Remember to credit everyone who participated in the research / investigation, unless they wish to remain anonymous – always check that in advance.

Closure (10 minutes)

Instructions

• Ask participants to create a takeaway poster by sharing their answers to the following question in the shared whiteboard:

  • What are your main takeaways from today's workshop?

• Give participants a few minutes to write their thoughts and read the thoughts of others.

Debriefing

• Highlight some of the points on the board.

Tools / Materials: No materials needed.

Instructions

• Wrap up the workshop and sum up its contents. Add any final words or highlight a main message.

• Run a quick review of the session. Each participants would say:

  • one thing they found very good about the session and

  • one thing they would improve for the next time

• Encourage participants to ask questions or give some final tips.

• Share contact information if relevant, and any follow-up details.

Contact Us

Please reach out to us at Exposing the Invisible if you:

• have any questions about this workshop plan and facilitation guidelines,
• use this workshop plan and want to share feedback and suggestions that can help to improve them,
• adapt the workshop plan to a specific context and want to share the results with us,
• want to suggest new activities, tips or examples that can be added to this workshop,
• want to share your expertise and collaborate with us on developing and testing new workshops.

Contact: eti@tacticaltech.org (GPG Key / fingerprint: BD30 C622 D030 FCF1 38EC C26D DD04 627E 1411 0C02).

Credits and Licensing

CC BY-SA 4.0

This content is produced by Tactical Tech's Exposing the Invisible project, and licensed under a Creative Commons Attribution-ShareAlike 4.0 International license

• Workshop authors: A. Hayder, Mario Rautner (as author of the "Evaluating Evidence" guide)
• Instructional design: A. Hayder
• Editorial and content: Christy Lange, Laura Ranca, Lieke Ploeger
• Graphic design: Yiorgos Bagakis
• Website development: Laurent Dellere, Saqib Sohail
• Project coordination and supervision: Christy Lange, Laura Ranca, Lieke Ploeger, Marek Tuszynski, Safa Ghnaim, Wael Eskandar

This resource has been developed as part of the Collaborative and Investigative Journalism Initiative (CIJI) co-funded by the European Commission under the Pilot Project: "Supporting investigative journalism and media freedom in the EU" (DG CONNECT).

This text reflects the author’s view and the Commission is not responsible for any use that may be made of the information it contains.