SENG 310 - Lecture 5
17 January 2015
Collecting and Analyzing Data
- Objectives
- Hypothesis
- Methods
- Conduct
- Synthesis
Collecting Data from Users
Important concerns when collecting data
- Study designs, pitfalls
- Ethics
Key techniques
- Interview
- Surveys
- Observation, Ethnography
- Web Analytics
Need to know what the questions are, need to be designed with what data needs to be collected in mind.
How will the data be analysed. How will the participants be required in an ethical manner.
Triangulate the data - Questions: Who, what where when, why, how. Have they been answered before?
Hypothesis
- Attitude-related hypothesis
- Behaviour-related hypothesis
- Feature-related hypothesis
Types of Data
- Video? Audio?
- Existing artifacts?
- Quantitative versus Qualitative...
- Challenges with different types of data?
- How will you analyze your data?
- When do you have enough?
Ethics
- Needs to be consent
- Inform about process, amount of time
- Participants should be over 18
- Should be no power-over relationships
- Should be no risk to participants
- Participants should be able to withdraw at any point
- Should inform participants if their data is anonymous or not
- Should be stored securely
- Should hide personal information
Interviews
- Unstructured - Open-ended questions (Can be insightful but not replicable)
- Structured - Tightly scripted (replicable but not open to emergent findings)
- Semi-Structured - Guided by a script and open ended questions (insightful and replicable, but hard to do)
- Focus Groups - Group interview (need to facilitator)
Interview guide:
- Introduce yourself
- Brief description and goal of the study and length of interview
- Basic demographics or factual questions (so you can interpret answers in context)
- A couple ice-breaking questions
- Key questions you care about
- How you will conclude, do they have questions, anything to add?
Example semi-structured interview questions:
- Which software tools do you use daily to communicate with others on your team?
- When is the last time you used these tools? Which features did you use?
- How did you learn about those features?
- Why do you find this tool easy (or hard) to use?
- “Tell me more about...”
- “Do you do that often...”
- “Why is that”
What not to ask or do
- Would some feature be useful to you? (bias)
- Why is the tool difficult to use?
- Avoid questions with yes/no answers
- Don't ask long or compound questions
- Avoid jargon
- Rush in to fill pauses
Surveys
- Can be given to many participants
- Closed questions are easy to analyze
- Need to pilot it several times
- Knowing how to sample can be difficult (avoiding biases)
- Question order matters
- Questions can be asked with bias
- Response rates will vary (20% to 40% is typically pretty good)
Survey Question Types
- True/False
- Multiple Choice
- Multiple Response
- Fill in the Blank
- Work Bank
- Matching Drag and Drop
- Matching Drop Down
- Sequence Drag and Drop
- Sequence Drop Down
- Numeric
- Hot Spot
Survey Pitfalls
- Failing to explain purpose
- Using jargon, long questions, too many questions
- Asking for sensitive information up front, failing to be anonymous
- Not showing progress indicators
- Failure to have some open-ended questions
- Having a survey that is too long
- Not sending out a reminder to complete it
- Failing to pilot it in advance
Observing Users
Direct: Can be in a controlled setting, or in the field
Indirect: Diary studies, interaction logging, video/audio trails
Ethnography
==Ethnography:== is observing people's behaviour in their own environments so that you can get a holistic understanding of their world - one that you can intuit on a deeper personal level.
- Ethnographers immerse themselves in the environment
- Can reveal the unknown, reveal context (semantics of living"
- Can be conducted over long periods of time
- Can lead to compelling solutions
- Online or virtual ethnography is another option (watch of ethics, deception!)
- Analysis can be difficult (too much data)
Assignments
None