Unexpected challenges, new opportunities, and an entirely new evaluation design: the evaluation work we’ve been doing this summer has taken numerous twists and turns as we adapt our initial design to the actuality of conditions here in Tbilisi.
We spent the spring semester designing an evaluation for the McLain Association for Children (MAC) as part of a project our 4-member team did for Dr. Heinrich’s program evaluation class at the LBJ School. We knew we were looking at the impact of MAC’s trainings for teachers of students with disabilities. We knew that our results would help MAC document the benefits of their work and make changes going forward. But our focus was very different.
We originally planned to evaluate the quality of Individualized Education Plans that teachers learn to write in MAC’s trainings. We wanted to evaluate the effect of these IEPs on children’s learning outcomes, but we knew this would be problematic due to a lack of a standardized measure of students’ levels. But we were going to try.
We found out very quickly once we landed in Tbilisi, that writing IEPs is only a small part of what MAC does. Interacting with actual people who had been the target group of the intervention helped us understand the goal of the intervention in a different light. As we talked with staff and interviewed training participants, we found that the real focus of the trainings is on the increase in knowledge of disabilities. Teacher after teacher told us they took the training in order to improve their ability to diagnose disabilities and to utilize the appropriate corresponding educational strategies to help each child develop and learn. Following our focus group discussions and interviews, we made changes in our initial design to shift the focus from IEP writing to knowledge transfer.
Another challenge that we ran into was the accessibility of institutions where we wanted to conduct the tests and focus group discussions. The Georgian educations system has been undergoing a comprehensive infrastructural reform since 2009. One result of these reforms has been a consolidation of some institutions and the closing down of inefficient institutions. So institutions we contacted were a little nervous. In addition, the position of school directors at public institutions is largely based on contacts and political affiliation. As the election season is approaching in Georgia, directors tend to be more on edge. They wanted to make sure that they do not allow any activity in their school that would in any way harm their reputation with the central government or that could provide the government with an excuse to close down the institution.
Some of the institutions flatly refused to meet with us. In some cases we were able to conduct tests solely because the MAC director used his personal contacts to assure the directors that we were not at all connected to the government. Once we gained access to the institutions there were even more challenges awaiting us.
The biggest challenge for us, in terms of data collection, was the collaborative culture of taking tests in Georgia. The education system in the country encourages collaboration even in individual assignments or at least the system does not enforce serious penalties for such activities. Despite the fact that we made it very clear to the respondents that the purpose of the tests was in no way to evaluate their performance, we had to use frequent reminders during the test that it is extremely important for us that they complete the tests on their own. It was quite the culture shock for us to watch teachers surreptitiously whispering answers to each other, or glancing over their colleagues’ shoulders, while completing the exams.
As we were to figure very soon, not knowing the native language was also a major hindrance in our information gathering exercise. Although we always had translators, it was very hard to connect with the focus group participants and we got a feeling that quite a lot could be lost in the translation. For our first two site visits, we only had access to a MAC psychologist as our translator. While this helped us immeasurably with gaining context as we visited sites, we worried that we were receiving only positive feedback as a result of her presence. MAC hired translators for our remaining visits, but we then lost some of the accuracy of the translation that we had gotten from her content expertise. (As an aside, we still received only positive feedback from participants, even when we specifically requested critiques.)
Also, if we ever had any hopes to learn the Georgian language, they were quickly dashed when we started practicing our new vocabulary in front of co-workers. The Georgian language contains many distinct sounds unlike any of the five languages that we collectively speak. Legend has it that ancient (and probably modern too) Georgian armies would ask people who were suspected of being spies to repeat certain Georgian words that contain these unique sounds. This was the ultimate test of Georgian nationality, failing which they would be punished for being spies from another country, particularly Russia. When we attempted to pass the test, our friends dissolved in uncontrollable giggles, erasing any hopes of reaching a level of respectable conversation before the end of the summer.
Fortunately, Georgian cuisine is much more approachable than the language. We quickly mastered the art of eating khinkali, dough dumplings most typically filled with pork, beef, cheese or mashed potatoes.
We also made sure to share our own cuisines with our co-workers. After a day of grappling with evaluation challenges together, it was important to find time to relax together as well.