Tuesday, April 14, 2015

Formative Questionnaire Design and Rationale


CHE142 Mid-Course Questionnaire

Dear CHE142 Student,
Congratulations! After several weeks of hard work, you have achieved many learning goals of the course and acquired new skill sets for your future career. Before we begin the second half of the semester, please take a moment to reflect on how you feel the course is going. Your feedback will allow me to customize the course content and adjust my instructions to meet your learning needs. Thank you.
Your CHE142 Instructor Tanya Tan
Please use the back of the page if you need additional space.
1. The topics that I have understood the best are:





2. The topics that I need further information or clarification are:





3. In terms of how the course materials are organized on Blackboard site, the aspects that I find the most helpful are: 





4. In terms of how the course materials are organized on Blackboard site, the improvements that I would like to see are:




5. The best aspects that I find in this course are:




6. I feel some aspects that could have been strengthened, such as:




7. The best aspects of the instructor’s teaching are:




8. The instructor will be more helpful if she/he can:




9. Additional comments for the course and/or the instructor:



(Optional) If you would like an individual follow-up from your instructor, you may write down your name below. (Otherwise, this questionnaire is completely anonymous.) Your instructor will contact you in 3-5 business days. Note this is optional.  _______________________________

Rationale

Formatting

The questionnaire is properly and professionally formatted:
·       12 pt. font with a standard font style Times New Roman that is consistently used in all other course materials
·       Standard margins with adequate amount of white space offering high readability and sufficient non-white space for information delivery
·       Pages numbered with descriptive header and last edited date[1]
·       All entries are numbered with sufficient lined room for comments. Instruction is clear that students can use the back of the page when additional space is needed.

Directions

Both written and verbal directions will be given. The directions are written in plain language, and revised to achieve the best clarity[2]. A first-person point of view is used in the questionnaire. The questionnaire thus holds a more personal, subjective, and natural tone of voice. It also allows the participant to directly recall their experience, personal thoughts, emotions and opinions in a natural way. The direction is comprehensive in a way that clearly communicates the purpose of the evaluation to the students (“Your feedback will allow me to customize the course content and adjust my instructions to meet your learning needs.”). It also acknowledges student’s learning and promotes reflective thinking. At the end of the questionnaire, students are given options to leave their names if they require individual follow-up. By doing so, instructors can reach out to solve more individual concerns that are not suitable to be addressed as a class.

Questions and Statements

               The questionnaire organizes questions in the following categories:
o   Student’s mastery level of course content: Question 1-2
o   LMS course site organization: Question 3-4
o   Student’s satisfaction: Question 5-6
o   Instructor’s Teaching: Question 7-8
o   Additional Comments: Question 9
Although the name of the category isn’t explicitly written as a subheading, it is apparent that all questions appear in pairs that inquire both positive and negative side of the same aspect. These categories of high concern provide me critical evidence to consider before the second half of the semester. Students can readily see the relevance of the questions, and are able to answer the questions. Questions are phrased in rather plain and clear language instead of obsolete terms. 
The purpose of this formative questionnaire is to spot check on learner’s progress and ask for individual student’s ideas, suggestions, and observations instead of numerical ratings. The class sizes range from 15-20 students which allows me to read every single word that the student has to say. If in the future, a formative questionnaire is collected from much larger class size, I will incorporate a proper evaluation scales for timely statistical analysis, and only provide 1-2 comment boxes for additional descriptive non-numerical comments.

Response fields

            As stated before, purpose of this formative questionnaire is to collect non-numerical information from the learner, check individual learner’s progress and understanding in a small-class setting. Comment areas are provided instead of numerical rating scale. Before implementing the questionnaire, I will also make the point clear that the more specific their comments are, the more accurately I can determine the next steps to take. Verbal and written directions also encourage students to use the back of the page as additional space. Analysis of the results of questionnaire require increasing time and effort. Bias and misinterpretation could occur.

Purpose

Skillful teaching is what helps students learn (Brookfield, 2006). Effective instruction is what helps students to learn in a more effective way that brings in changes in knowledge, skill set and mind set. To achieve instructional effectiveness, both instructors and students need to develop a clear understanding of learning outcomes and frequently examine progress. The first two questions (topics that students understand the best, and topics students need further clarification) focus on this characteristic of instructional effectiveness. They reveal how learners feel about knowledge acquisition. Consecutive instructions become directional to help students achieve the unaccomplished learning outcomes.
As a blended course, usability and ease of navigation are important evaluands, especially when a learning management system is new to learners (e.g. freshmen students). Failure to locate information drastically hinders learning. Hence, question 3 and 4 are designed to survey student’s user experience of the Blackboard site. In the first two weeks, I often ask students whether they feel comfortable to obtain information from Blackboard site. I also demonstrate how materials are organized online. In the midpoint of the semester, I am also interested in how comfortable students are with the online portion of the course.
Question 5 and 6 are where students address their feelings. They are more of a general inquiry of student’s satisfaction. Effective instruction requires meaningful student-student, student-faculty and student-content interaction. Students’ satisfactions level is critical information to assess instruction effectiveness. 
Question 7 and 8 evaluate instructor’s teaching strategies and quality of presentation. The five main competency area for instructors include: professional foundations, planning and preparation, instructional methods and strategies, assessment and evaluation, and management (The International Board of Standards for Training, 2003). For online and blended course instructors, online teaching competencies emphasize on technical skills and skills needed to facilitate online discussion. Online instructor's competencies involve[3] (Spector, 2001)
·        Allowing learners time for reflection
  • Keeping discussions alive and on a productive path
  • Archiving and organizing discussions to be used in subsequent lessons
  • Animating discussions and displaying cultural sensitivity
Patti (2004) summarized additional online instructor's competencies. These are also relevant to blended course instructors.
  • Administrative - to assure smooth course operation and reduce overload
  • Designing - to assure adequate eLearning outcomes and satisfaction
  • Facilitation - to provide social benefits and enhance learning
  • Evaluation - to assure that learners know how they are evaluated; and help learners meet the course objectives
  • Technical - assure overcome barriers due to technical components
When implementing the questionnaire, I will lead the students to carefully recall their feelings to presentations, designed activities, and assessment. As a formative questionnaire, question 7 and 8 are phrased rather general to let the students express what they like and dislike. In all five main areas for instructor competency (The International Board of Standards for Training, 2003), students are both participants and observers who can often pinpoint what has been done properly, and what hasn’t. For example, it is difficult for the instructor to find a misalignment between assessment and instruction when everything appears to be “natural” to us. As a novice learner, they may come in with no prior knowledge in a certain field – all they know is what has been taught. This makes their responses valuable input.  

Implementation and Review of Results

The original version of the questionnaire was implemented in the classroom two weeks ago. I chose the class that I have the least materials to teach, and gave students 20 minutes to complete the questionnaire. I announced the purpose of the questionnaire and explained directions (such as specific comments, things and experience to reflect on). I also made the point clear that they do not have to answer all the questions; however, anything that they would like to say will be thoughtfully examined and greatly appreciated. I also informed the students when they could expect hearing back from me regarding the results of the questionnaire. A volunteer was chosen from the students to administrate the entire process when I stayed outside of the classroom.
65 complete questionnaires from four sections were collected. Within each section, similar results were grouped and responded together. My response is created in a personal and friendly tone that articulates the following:
·        Acknowledgement of student’s progress
·        Appreciation of suggestions and comments
·        Changes that I decide to make
·        Changes that I am not able to implement, why and what alternate solution is
The next week, I let the students vote how they would like to know my answers, either communicated verbally, or posting the questionnaire and my answer on the wall so everyone can look at, or posting a summary on Blackboard that they can view online afterwards. The vote was quickly administrated by a volunteer. All the students are more comfortable with an open wall of questionnaire and responses. I kept onymous questionnaire not published (students can choose to leave their name on the questionnaire if they require individual follow-up), and then offered sufficient time for students to read the questionnaire and my feedback. It appeared to be a favored approach as I already grouped the similar questionnaire results. Confidentiality issue was one of my concerns, so I let the students decide how they would like to know the results.  
Learners often report a feeling of impostership – they lack the intelligence and talent to succeed. A way to deal with this impostership is to make this phenomenon public (Brookfield, 2006). By publishing the questionnaire results, it is potentially useful to let imposter-feeling students know that they are not alone. There are common difficult topics that many of the students struggle with. I let them know specific studying advice, and what I plan to do to help them overcome the barriers. Three out of sixty-five students wrote their names on the questionnaire requesting a confidential follow-up. I emailed them and scheduled meeting time within the time that I promised.
I am also planning on sharing the questionnaire results, my interpretation and responses to my mentor. I would also seek advice and sharing of his teaching experience in the previous years. In the future, I will conduct a pilot test with students and revise questions accordingly before large scale implementation.

References

Brookfield, S. (2006). The skillful teacher: on technique, trust and responsiveness in the classroom, 2nd Edition. San Francisco, CA: Jossey-Bass.
Shank, P. (2004).  Competencies for Online Instructor. Retrieved from http://m.kettering.edu/sites/default/files/resource-file-download/CompetenciesOnlineInstructors.pdf
Spector, J. M. (2001). Competencies for online teaching. ERIC Digest, 1-11.
The International Board of Standards for Training, P. a. (2003). Instructor Competencies. Retrieved from http://www.ibstpi.org


[1] Part A Questionnaire is incorporated in this assignment with the original header removed.
[2] The author greatly appreciate revision suggestions from PIDP3260 classmates Laura Lotzer, Tanya Allen, Andrea Kettering and Martha Yu. 
[3] As posted in Moodle discussion

No comments:

Post a Comment