Author Archives: achilleshakeem

About achilleshakeem

Jenny Achilles is an MPAff student at the LBJ School of Public Affairs. Her academic interests include non-profit management and social policy. Her international expertise focuses on Latin America. She has worked as a newspaper reporter and as a study abroad advisor prior to joining the LBJ School and holds a master’s in journalism from UT.

Ahmed Bilal Hakeem is also an MPAff student at the LBJ School of Public Affairs. His academic interests include international development and education policy. His non-academic interests include introducing cricket and Pakistani food in the Caucauses. Prior to joining the LBJ School, he worked at a microfinance organization in Pakistan.

Challenges in the field

Unexpected challenges, new opportunities, and an entirely new evaluation design: the evaluation work we’ve been doing this summer has taken numerous twists and turns as we adapt our initial design to the actuality of conditions here in Tbilisi.

We spent the spring semester designing an evaluation for the McLain Association for Children (MAC) as part of a project our 4-member team did for Dr. Heinrich’s program evaluation class at the LBJ School. We knew we were looking at the impact of MAC’s trainings for teachers of students with disabilities. We knew that our results would help MAC document the benefits of their work and make changes going forward. But our focus was very different.

We originally planned to evaluate the quality of Individualized Education Plans that teachers learn to write in MAC’s trainings. We wanted to evaluate the effect of these IEPs on children’s learning outcomes, but we knew this would be problematic due to a lack of a standardized measure of students’ levels. But we were going to try.

We found out very quickly once we landed in Tbilisi, that writing IEPs is only a small part of what MAC does. Interacting with actual people who had been the target group of the intervention helped us understand the goal of the intervention in a different light. As we talked with staff and interviewed training participants, we found that the real focus of the trainings is on the increase in knowledge of disabilities. Teacher after teacher told us they took the training in order to improve their ability to diagnose disabilities and to utilize the appropriate corresponding educational strategies to help each child develop and learn. Following our focus group discussions and interviews, we made changes in our initial design to shift the focus from IEP writing to knowledge transfer.

Returning from a site visit where we tested for knowledge of disabilities.

Another challenge that we ran into was the accessibility of institutions where we wanted to conduct the tests and focus group discussions. The Georgian educations system has been undergoing a comprehensive infrastructural reform since 2009. One result of these reforms has been a consolidation of some institutions and the closing down of inefficient institutions. So institutions we contacted were a little nervous. In addition, the position of school directors at public institutions is largely based on contacts and political affiliation. As the election season is approaching in Georgia, directors tend to be more on edge. They wanted to make sure that they do not allow any activity in their school that would in any way harm their reputation with the central government or that could provide the government with an excuse to close down the institution.

Some of the institutions flatly refused to meet with us. In some cases we were able to conduct tests solely because the MAC director used his personal contacts to assure the directors that we were not at all connected to the government. Once we gained access to the institutions there were even more challenges awaiting us.

The biggest challenge for us, in terms of data collection, was the collaborative culture of taking tests in Georgia. The education system in the country encourages collaboration even in individual assignments or at least the system does not enforce serious penalties for such activities. Despite the fact that we made it very clear to the respondents that the purpose of the tests was in no way to evaluate their performance, we had to use frequent reminders during the test that it is extremely important for us that they complete the tests on their own. It was quite the culture shock for us to watch teachers surreptitiously whispering answers to each other, or glancing over their colleagues’ shoulders, while completing the exams.

As we were to figure very soon, not knowing the native language was also a major hindrance in our information gathering exercise. Although we always had translators, it was very hard to connect with the focus group participants and we got a feeling that quite a lot could be lost in the translation. For our first two site visits, we only had access to a MAC psychologist as our translator. While this helped us immeasurably with gaining context as we visited sites, we worried that we were receiving only positive feedback as a result of her presence. MAC hired translators for our remaining visits, but we then lost some of the accuracy of the translation that we had gotten from her content expertise. (As an aside, we still received only positive feedback from participants, even when we specifically requested critiques.)

Also, if we ever had any hopes to learn the Georgian language, they were quickly dashed when we started practicing our new vocabulary in front of co-workers. The Georgian language contains many distinct sounds unlike any of the five languages that we collectively speak. Legend has it that ancient (and probably modern too) Georgian armies would ask people who were suspected of being spies to repeat certain Georgian words that contain these unique sounds. This was the ultimate test of Georgian nationality, failing which they would be punished for being spies from another country, particularly Russia. When we attempted to pass the test, our friends dissolved in uncontrollable giggles, erasing any hopes of reaching a level of respectable conversation before the end of the summer.

We enjoy a lunch of mushroom khinkali at a Tbilisi restaurant.

Fortunately, Georgian cuisine is much more approachable than the language. We quickly mastered the art of eating khinkali, dough dumplings most typically filled with pork, beef, cheese or mashed potatoes.

We also made sure to share our own cuisines with our co-workers. After a day of grappling with evaluation challenges together, it was important to find time to relax together as well.

We made Pakistani daal and Texas chili to thank our co-workers.

Changing attitudes toward disabilities in Georgia

We wanted to explain a bit about why our project is so important. The McLain Association for Children (MAC) trains professionals who work with individuals with disabilities—primarily with children—how to properly diagnose and work with those individuals.

MAC'S logo

MAC's logo

Prior to last semester, we were not familiar with the status of education for children with disabilities in Georgia. What we have learned in the past few months is that best practices are developing but much still needs to be improved.

In many cases, children with disabilities are in institutions were they are merely being babysat. Many times there is no goal for educational progress. When teachers are trying to work with children, they do not have the knowledge and resources necessary to best help the children learn. It is only recently that the Ministry of Education in Georgia developed a certification for caregivers and teachers working with individuals with disabilities. In our focus group discussion, most caregivers stated that their primary sources of knowledge regarding disabilities were either “general knowledge” or senior colleagues as opposed to any formal training or educational program.

In many situations, the attitude of the parents does not help with child development. Due to the stigma attached with disability, parents do not help their children in integrating with the rest of the society. At times, these children are considered as a social and financial burden on the family. These attitudes severely inhibit the children with disabilities from learning and developing in such an environment.

The goals of MAC’s trainings have several facets. They want to improve teacher and parent attitudes toward children. They want them to understand different diagnoses and how this affects how children learn and mature. They want the teachers to employ strategies for helping children learn and develop to their full potential.

It can be hard to visualize what this means. We have begun conducting focus groups with teachers, and we are beginning to understand better the impact of MAC’s trainings. For example, in many instances, teachers previously thought that students were just acting up or being “naughty.” After the trainings they learned that many of these behaviors are directly tied to the child’s disability. They leave with strategies for engaging the child in learning activities in a way that minimizes these “problem” behaviors.

A second example: at one site a director described a new warmth in the manner in which teachers treat children. They smile more now that they understand what children’s diagnoses and symptoms are. This can help with the overall atmosphere and can lead to improved learning outcomes for the child as well. Imagine the difference it can make if a teacher is no longer frustrated with a child’s misbehavior, but is rather smiling and encouraging the child to find active learning activities!

In addition, MAC brings best-practices in resources. Their occupational therapists can train children how to correctly sit in wheelchairs so that their bodies develop correctly in spite of the chair. The group also helps make tactile tools for child development. We heard stories of autistic children who had been aggressive and frustrated. Then one day they were presented with a series of sensory buckets—cold water, hot water, dough—and the new sensations brought smiles to their faces. Similarly we met a young man who had been paralyzed in the lower body and was living on the 6th story of an apartment building that did not have ramps or elevators. He had taught himself fluent English and was now planning to learn German. MAC met with him a couple of years ago and after evaluating him, provided him with a laptop. With the help of a laptop, Internet, like minded-friends and a winning attitude, he started an NGO for educating individuals with disabilities. In addition, he also recently graduated from his freshman year in an online degree that he is pursuing at a university in the U.S. It was amazing to see how encouragement and positive attitudes can change lives. Story after story of how simple techniques can make a significant difference in the quality of life for children and for caregivers have underscored for us how important this work is.

In the U.S. we have learned through years of campaigns by dedicated professionals to expect that individuals with disabilities can be integrated into society as a whole. We expect them to be able to function normally and as a society we make physical accommodations to help facilitate their rights. Individuals with disabilities in many other countries do not have access to physical accommodations. Navigating most streets and buildings in Georgia would be impossible for a person in a wheelchair, for example.

Stereotypes and stigmas of disabilities mean that many children are ignored and shut away during key developmental years. They are not taught independent skills that can lead to a happier, more productive future.

We are excited to be learning about the changes taking place in Georgia and to be providing a written evaluation for MAC so they can continue to enhance and adapt their trainings to bring change in knowledge and attitudes about disabilities and to empower Georgians with disabilities to reach their full potential to learn and lead as independent lives as each individual has the ability.

Summer Internship with the McLain Association for Children (MAC) in the Republic of Georgia

We are interning with the McLain Association for Children (MAC) in the Republic of Georgia.

feast table

We enjoyed many a Georgian feast while we were recovering from Jetlag in the village of Dzevry.

While we both have international policy interests, neither of us has regional expertise in the Caucuses. We chose this internship due to the importance of disability as a development issue in the region and the lack of prior research on this subject. This internship will give us an opportunity to learn, share our knowledge and contribute to the development of disability policy in the region.

We began work on this as part of our semester-long project in the Program Evaluation class with Dr. Carolyn Heinrich. Our four-person team designed an evaluation for one component of MAC’s work. We traveled to Georgia to refine the design, to expand the scope to other components, and to begin implementation.

During our first week here we first spent time recovering from jetlag and observing Georgian culture in a laid-back manner. We spent four days in a village 3 hours from Tbilisi, toured a facility that will soon begin working with MAC, and ate, and ate, and ate.

We are interviewing the director of one of the institutions MAC has trained.  Nino Lomidze (right), a psychologist with MAC, translates the interview.

We are interviewing the director of one of the institutions MAC has trained. Nino Lomidze (right), a psychologist with MAC, translates the interview.

As we begin our formal workdays, we have outlined our goals for the summer. We will be producing an evaluation report by the time we leave. Our focus will be on process evaluation of the implementation of MAC’s trainings at the different institutions they work with. Of the 27 locations where they have conducted trainings, our goal is to conduct in-depth evaluation of nine. We will be conducting focus groups and key informant interviews in order to obtain the opinions of trainees as to what has been most useful for them. We will also be able to get a better picture of what elements of training they are implementing on a regular basis. In addition, we will administer a second round of the post-tests that MAC uses to test the participants’ knowledge at the end of training. We want to see how much knowledge they are retaining. Moreover, we plan to gather information from the rest of the institutions through survey questionnaires.  We will also run some basic descriptive statistics of the results of the original post-tests for all 27 institutions.

Jenny and Bilal with MAC co-founder Roy Southworth (left),  fellow intern Margo Poda (center) and English teacher Ernest (right). In the background are cave dwellings from centuries ago.

Jenny and Bilal with MAC co-founder Roy Southworth (left), fellow intern Margo Poda (center) and English teacher Ernest (right). In the background are cave dwellings from centuries ago.

Professionally, we are most looking forward to seeing how our theoretical evaluation design from class works when we implement it in the actual situation it was designed for. We are also very excited to have an active part in MAC’s work to bring best practices to Georgia in education and care for individuals with disabilities. This evaluation study is going to serve as an important resource for other institutions in the region with similar programs. The results of this evaluation study can be used for enriching the educational policy discourse in the region and developing evidence-based educational and health intervention programs for community members with disabilities.

Neither of us had much experience with education of children with disabilities in our own countries. We have learned much already about this important sector of education. It is also fascinating to learn about Georgian culture in the context of education for children with disabilities.