Episode Transcript
[00:00:06] Speaker A: Welcome to the Leadership window podcast with Dr. Patrick Jenks. Each week through a social sector lens, Patrick interviews leaders and experts and puts us in touch with trends and tips for leading effectively. Patrick is a board certified executive coach, a member of the Forbes Coaches Council, a best selling author, award winning photographer and a professional speaker.
And now, here's Dr. Patrick Jenks.
[00:00:31] Speaker B: Hey everyone. Welcome to episode 159 of the Leadership Window. Glad you're along. We're going to kind of nerd out today, as I call it.
My guest today is Dr. Beth DiPonte.
She is the founder and leader of Social Science Consultants, which has provided evaluation, demographic research and community assessment services for more than 13 years, working with a global network of social scientists. The Firm supports nonprofits, NGOs, governments, international organizations and more through program evaluation and monitoring, demographic analysis, survey design, and all kinds of other data driven approaches to understanding impact.
Beth is the author of Evaluation Essentials. That's a book published by Wiley and Sons and it's in its second edition which was released last year. Her careers included serving as Chief of Section with the United Nations Office of Internal Oversight Services, Investigation and Evaluation Division, as well as faculty roles at Yale, Carnegie Mellon University of Pittsburgh. She currently lectures at Yale's School of Management and she's authored numerous UN General assembly reports and peer reviewed academic articles in the areas of demographic methods, human rights, and even US Food assistance policy to get that specific. So for more information on Beth and what all she's doing and how she can help you or your organization, visit socialscienceconsultants.com the link is in our show notes and on our page. And Beth, I feel like I just introduced about four different people just now. Did I miss anything? Welcome to the Leadership Window.
[00:02:27] Speaker A: Thank you so much for having me, Patrick. And thank you for that generous introduction. And yeah, I think you did introduce different people and I think that represents how I've aged and how I've grown throughout my career, starting from a very quantitative social scientist to becoming far more interested in qualitative methods and merging methods. And yeah, that's what I like to
[00:02:55] Speaker B: do best, merging methods is that, that's not a term I'm familiar with. Is that the same as mixed methods?
[00:03:03] Speaker A: Mixed methods triangulation?
Yeah. Yeah.
[00:03:06] Speaker B: Okay. I thought that might be the case. But yeah, you know, so you and I both have PhDs I want to learn about. Yours in mine was a qualitative study and I, you know, as much as I value the quantitative, I just.
Well, first of all, I didn't think I could do It.
But actually I did want to. But my program was condensed. It was a. It was one of these accelerated programs programs, and we didn't really have time to build out the quantitative piece of it. But we'll talk a little bit about that. What was your PhD in?
[00:03:43] Speaker A: My PhD is in sociology. I focused on demography, on demographic methods. I was in the Committee on Demographic Training at the University of Chicago as a recipient of the NICHD Demographic Fellowship. As an undergrad, I studied math and sociology, which is kind of an odd mixture.
I had a mother who grew up in abject poverty, want her daughters to only study sociology because she thought we could never have a. That I would never have a career. And so she kind of insisted that we also study math. I have a sister who also majored in math, and she thought that with a math major we could always get a job, we could always become a math teacher. Well, when I went to grad school, I decided to. I was passionate about sociology.
And when I went to the University of Chicago for grad school, they saw a lot of professors, saw my math and sociology degrees, and I was kind of a natural fit for demography, which I love. I love the field of demography. It's fascinating, but I've grown from there. And then I got more into program evaluation.
[00:04:57] Speaker B: Tell me, what's the love of demography? What is it that inspires you about that? I'm truly interested. Yeah.
[00:05:03] Speaker A: Yeah. Well, they say demographics, that determines so much. It determines politics, it determines labor forces, you know, and the field of demography incorporates a lot of different aspects. I mean, there's economic demography, social demography, asking questions about who we marry, when do we marry, when do we have babies, when do we die? And my field, my real specialty became mortality.
Looking at life tables and looking at death rates, that tells you a lot about the social situation, situation of a population.
Looking at. Looking at various mortality rates and schedules of mortality, which is. I mean, to me that's fascinating. I kind of merged. I morphed into program evaluation, more or less with my work on the first Gulf War, which was looking at what the mortality effects were from the Gulf War. So I was a demographer that was thinking the program, the intervention, was war.
Then you can look at, as an outcome measure, you can look at mortality rates and how it affected the life tables of people.
[00:06:24] Speaker B: Wow, that is fascinating. And you kind of answered. I was wondering what led you. I'm always interested in what leads people to their work. I mean, there's relatively, I guess not a whole lot of people that geek out over this stuff. Right.
So what led you to this? Where did this interest of sociology and demography and data science, Science and evaluation, where did all this come from?
[00:06:49] Speaker A: I think some of it came from growing up in a town in Connecticut that at the time, when I grew up in the 70s and 80s was very much a factory town which was going, which underwent a lot of transitions.
The manufacturing base declined somewhat, Factories, some factories closed. There still is, there still are a few factories in town. The biggest, of course, is Sikorsky aircraft, which makes helicopters.
But seeing those changes and how they affected people at different.
The town I come from, Stratford, Connecticut, it's a really heterogeneous town in all ways.
Economically, racially.
Yeah, so it was quite the melting pot. And so coming from that sort of environment, I was always interested in social forces, essentially. And so my dissertation ended up being on the Puerto Rican population.
At the time, people were talking about an underclass and the Puerto Ricans, the Puerto Rican population, especially in the New York area, being part of the underclass. It's a term that's not used anymore. But there was a real question that I had of what, what happened to this population which was so dynamic and, and had really good economic prospects in the 40s and 50s and 60s. And then kind of the, the floor fell out and the floor fell out from them. And essentially, I think it was a story of women's labor force participation. They had really high labor force, female labor force participation rates in the 50s. And in the garment industry, the garment industry left New York, went down south.
Part of the reason it went down south was because there weren't the unions to contend with and lower wages could be offered. And essentially a lot of Puerto Rican women left the labor force. They didn't have opportunities because they didn't necessarily have language skills needed for the pink collar industries that were evolving in New York City at the time.
So to me, that was a really interesting question of how does women's labor force participation contribute to declines in the economic well being of an ethnic group?
[00:09:29] Speaker B: Man, that is very fascinating. And for data, people that look into things like variables and correlations and things you're talking about, you know, sort of moderating variables, mediating variables, you know, the, the covariance, you know, you look at, you look at a whole population like Puerto Rico, okay, well then you break it down further and further and further and you learn that there are certain social elements, determinants, whatever you want to call them, behaviors that for seemingly the same population, one little variable One little difference changes the whole game for a subpopulation. And that's what you're talking about. It's really interesting. You're making me think, Beth, of. I don't know why this just came to mind, but this isn't just about. I mean, I don't think it is for you, based on your career and what you're doing to help organizations now.
I don't think this is just about. Ooh, I just think this stuff is cool.
There's a purpose for data, right? It informs us so that we can improve things, in this case, social fabric.
And you're reminding me of a series of questions, a sequence of questions asked by Warren Berger. And I'm not talking about the old Supreme Court Justice, I'm talking about the author of a book called A More Beautiful Question.
And he talked about the sequence to innovation being three questions. Why, what if? And how or how might we.
And you know, the data, in looking at doing research and doing analysis is asking why, right? Why is a certain population dying earlier than others? Why do some people get married earlier in life? Why is this population, you know, not graduating at the same rate as others?
And so the first question is why? The second question is, what if?
What if we could open this door for that population? What if we could add this? What if we could tweak that?
How might we do that?
And that really is program evaluation in a nutshell, now that I think about it. It's that kind of innovation. What do you think about that sequence?
[00:11:54] Speaker A: No, that's absolutely correct. It's so insight.
The why question is asking historically, right, what happened? What's the story?
The what if is what could we have? What can we change going forward that will benefit all or at least the most people, right?
But you, you need to understand the why first. And so doing that really careful analysis of data, quantitative data, qualitative, asking people like, what happened? Does my understanding of the quantitative, does that gel people's understanding of the situation?
If you see that, I mean, we call it mixed methods, right? And if you see that gelling, that's great, then you're probably on the right track. If you don't, then you have to start asking other questions, like, are people remembering wrong? Are the data collection approaches not including the entire population, which is always a concern of mine, like, we could have data. That doesn't mean it's good data or it's representative data. And so with any data, you also have to ask who is included and who isn't included in various data sets. How did they get into the data set to begin with?
[00:13:15] Speaker B: I think the other thing is that even with data, if it's incomplete data, we make assumptions about what's going on. We create a correlation that doesn't exist. I remember in a small town in Virginia where I used to work in the United Way world, we were looking at early childhood. We were looking at kindergarten readiness rates.
And demographically, we found what we wanted to do is we wanted to use Chip and Dan Heath's model in the book Switch, where we find the bright spots and replicate those, rather than go, well, what's wrong with this center? And why are these kids not ready? We wanted to find out what kids are ready and what's different about them.
And one of the things that we found at the time, it was a great learning moment for me was that there was a.
There was a school, a public school, who had an early childhood element of their. Of their school.
And I don't. I don't. I think. I don't think it was a Head Start, but it might have been a Head Start program.
But this was in a school where typically many people would make assumptions about the population.
It's very diverse, heavy African American, as a matter of fact, but in a note, wealthy part of town.
And you would look and you would assume. Many people would assume, oh, the kindergarten readiness rates there are probably a lot lower. What we found was that out of this one school in this area where we were making demographic assumptions, they were actually getting 90 to 95% kindergarten readiness. And we went in to find out, what are you doing differently? Because obviously, it's not just the stereotypical things you think, right? Where white kids are performing higher than black kids or brown kids. There was something very different that this school was doing that really led us to be able to replicate that on the north part of town. And they saw in. In three to four years, kindergarten readiness rates go up in more areas because of applying certain behaviors and techniques that we're doing.
Long story. To just say, you know, data is good, but you got to make sure it's the right data and that it's telling you the full story. Yeah.
[00:15:39] Speaker A: So you have to tell me, what were the techniques?
[00:15:42] Speaker B: Well, some of it was parent education. A lot of it was parent engagement and education.
Some of it was in curriculum. They had quality faculty, you know, rated. You know, they had all the five star ratings of what they were doing and just some of the curriculum and what they were doing in their early childhood program that we just went in and looked at. And we had a director of a Smart Beginnings program at the time who could tell you more about it than I can.
But they were able to find three or four bright spots, just like in the book Switch, if you've read that, and where. The researcher went to Vietnam to find out about this malnutrition in these villages. And he found one village with the same demographics.
They were doing two simple things different. They were adding shrimp to their rice for some more protein, and they were changing the scheduling for eating. Rather than having one big meal in the day, they were splitting their meals up throughout the day. And that's all they were doing differently.
So it was, you know, again, a case where data looks at the bright spots. Not just, you know, why is this one failing? It's also, why is this one succeeding? What's different between them?
[00:17:01] Speaker A: But both of those examples are really good examples about how just having quantitative data wasn't going to tell you the story.
You really need to go in and understand the situation. You need to actually look at the curriculum, see how teachers were engaging with the young children.
You need to go into the villages in Vietnam and ask them really detailed questions about what is.
Like, how did the eating schedule even come be?
[00:17:32] Speaker B: Just observation.
Yeah, I guess. You know, that's an interesting point, Beth. I hadn't thought about this. Maybe I'm gonna say something that might not be right. And all the real data people out there, like you were gonna go, no, no, no. You don't know what you're talking about. But what you're describing sounds like quantitative data is good at telling you that it's happening.
Qualitative data is probably better at telling you why it's happening. Happening.
[00:17:56] Speaker A: Yes, I think that's right. Well, quantitative data is also better. You know, I always tell clients and students, I tell them, you know, you don't do a survey to understand what is happening. You do a survey to understand the prevalence of what you think is happening.
It's about prevalence. It's about getting statistics.
So, yeah, so quantitative data has its. Has. Definitely has its limitations.
And I think what it may be getting lost in education, though, is this mixture of the quantitative and qualitative and the appreciation of having to tell the whole story, not just the quantitative story.
[00:18:47] Speaker B: So tell me more about your work. When we say program evaluation, this is not organizational auditing. This is. No, this is down into, you know, a social impact organization who has a program, whatever that might be. If it's a homelessness sheltering program or if it's a early childhood development Program or whatever it is, it's about evaluating the effectiveness of our solution, our application to the issue.
And is that the same thing? You know, in, in the United Way world, we grew up learning the term outcome measurement, and I do a lot of teaching of that. Is that the same thing or is evaluation something deeper than outcome measurement?
[00:19:33] Speaker A: Well, I'd say broader. I wouldn't necessarily. I mean, outcomes can be one measure of impact, you know, looking at change in outcomes over time, you know, before and after having some sort of quasi experimental design where you are using as your observ particular outcomes, but outcomes themselves aren't going to answer the question of how relevant was the project. Was this the right approach for this population? Was this the best approach for the population? That's not an outcome question.
So, you know, the relevance question is very important, you know, and outcomes aren't necessarily going to tell you was this the best way to use the money and resources for this problem or for this population?
[00:20:23] Speaker B: Yeah, so that's good. The outcome.
[00:20:25] Speaker A: Outcomes have its limit. Outcomes are, you know, they're measures.
And I always tell people, think about, like your outcomes in words before you think about outcome measurements, because outcome measures, that's a whole different discussion about what sort of data are available and. Yeah.
[00:20:44] Speaker B: What are you trying to achieve with the people you're serving or for the people you're serving?
Yeah. One way that we teach it when we're training in outcome measurement is what's what changed as a result of your work? So your work, you know, you can look at the quality of your work, the performance, the outputs, how much of it you're doing, how satisfied everyone is, how good a time they had all those things. The question is, what changed as a result?
[00:21:10] Speaker A: That's. Yeah, that's somewhat accurate. I mean, evaluation also asks questions about relevance. Right. Is this the best use of funds? Is this the best intervention for the problem at hand?
[00:21:23] Speaker B: Yeah.
[00:21:24] Speaker A: So if you restrict your analysis to only the intervention at hand, and let's say you, you're an evaluator and you're looking at the intervention that a client has, I also think it's your responsibility for the client to do a literature review to ask the question of, to answer the question of how have others dealt with the same problem?
Because they should know how their intervention fits into the whole broad scheme of approaches to a problem.
[00:21:54] Speaker B: That's a really good point, Beth, because, you know, in school, when you're doing a thesis or a dissertation, literature review is a big part of it, and that is going out and seeing what research has already been done, what's already known, what's the current body of knowledge out there and what a great approach or method to use in our work is to go out and study and learn what's going on out there. I will say one of the things that I've said for years in my leadership coaching world is, you know, people say, yeah, you know, I need to do more reading, I need to do more learning, I need to become more of an expert in my field. I need to. But I just, you know, I don't, I don't really have the time because when I'm at home I'm doing this, that or the other.
[00:22:38] Speaker A: Yeah.
[00:22:38] Speaker B: And I've always told them, in my opinion, research and learning are not luxurious to your job, they're part of your job.
That's work time, you know, to learn these things. And that includes going out and finding what's working and what's not working.
[00:22:57] Speaker A: Yeah, yeah. And I think that's one of the things. I'm always surprised that people who work in an area for 40 hours a week for 10 years haven't looked at the literature on that particular area.
I'm really surprised at that. And I think one of the things that evaluators bring to the table, evaluators who do strong literature reviews, is sharing that knowledge of how others have dealt with problems with the same sort of challenges.
And that's really eye opening to clients at presentations. Clients are always, I mean, I can just see the light bulbs going off in their heads about, you know, what if we had tried this instead of what we're doing? Or maybe we can supplement the activities that we have now with these other activities. But learning from other organizations is extremely important, I think, and it's a really good use of resources. It doesn't cost that much to do a literature review in comparison with what the size of programs are and what the expenditures are on them. But they can really serve as a basis for clarifying programs, for improving them, for showing what the best case scenarios could be.
So it's a very good use of money to invest it, to have other people do a literature review. I can understand implementers saying that they don't have the time to read all the literature. So that's. And they may not even have access to it because a lot of, a lot of articles are behind paywalls that are expensive.
They're journal articles. You know, they may not, implementers may not have access to those, to that sort of information, but evaluators should have access to that information, and that's a critical part of any evaluation. I always. I start every evaluation I do off with a literature review.
[00:24:59] Speaker B: Well, I'll put in a quick plug for local libraries.
[00:25:03] Speaker A: Yeah, local libraries are great resources and they often do have the access, at least some journal access, depending on.
[00:25:11] Speaker B: Depending on your library, depending on the library. Some of them have pretty extensive journal access. I mean, the big databases that hold a number of journals on management and social science and all kinds of things. It might not get you everything, but there's a lot you can get for free and there's a lot of your own research that you can get for free. One of the things I'm finding, Beth, in my work of coaching.
Coaching is still a developing field. It's not regulated, but it is becoming more and more professionalized through, you know, the center for Credentialing and Education or through the International Coach Federation and other groups like this.
But right now, one of the things we're struggling with in our industry is that that longitudinal study is still pretty weak in the industry as to how coaching impacts organizations. There's some stuff out there, but it's still fairly young.
This is not the case for most of the social impact work I see being done in the nonprofit sector. There's a ton of longitudinal research on housing and homelessness, on food insecurity, on early childhood development and, you know, I mean, domestic violence. You name it. There's a lot of it out there.
[00:26:32] Speaker A: Yes, yes.
[00:26:33] Speaker B: So we do need to take advantage of it. I had Doug Paul on, on the show a few weeks ago talking about his impact operating system. So this eos, this entrepreneurial operating system is pretty trendy right now. A lot of organizations looking at implementing off the book traction, if any of you are familiar with this.
And a lot of the operating system stuff is about measuring inputs and outputs. It's some, many of it is the business aspect of it. It's operating. But there's a part in Doug's impact operating system for metrics around evaluating programs and mission work.
And it dawned on me as I was talking with him that evaluation is kind of the R and D of the social sector. It's. It's research and development, just like you would a product, testing a product, finding out what customers like about it, use about it, don't use about why they buy it, what they would buy if you had it. That's the same thing really with program evaluation is you're researching what's possible. Then you're researching, you're developing your own projects or programs.
As you evaluate those programs, you're learning what's working, what's not working, and you develop it further. I mean, that just rang to me as if program evaluation in a nonprofit is like R D in a corporation.
[00:28:02] Speaker A: Yes and no. That's a really good point.
And that's true for pilot projects. Right. For projects where you have a theory of change, it hasn't been implemented elsewhere, or you're trying something new on a well developed theory of change and you want to see how that's going. But you know, another aspect of evaluation, depending on what you're evaluating, you could be evaluating a project that has a very well developed theory of change. It's been implemented all over the place and you're in the focus of the evaluation ends up being more implementation.
Sometimes those evaluations look more like automatic audits, performance audits.
You know, they aren't really questioning, they may question the relevance, but they aren't. I mean, they really want to know what was done, what was done well, how it could, could have been implemented better, whether it's, you know, they may be benchmarking it with similar initiatives.
[00:29:07] Speaker B: That's a really good distinction. Yeah, what, what you're evaluating.
And they're, they're different kinds of data and they're just different ways of looking at them. That's a really good distinction.
[00:29:18] Speaker A: You.
[00:29:18] Speaker B: So I'll let our listeners in on a little bit of a connection here. You've done some work with an organization up in Frederick, Maryland that I've also done some work with. And I'll call them out, we'll give them a plug because they're an amazing organization. They're called Second Chances Garage.
And basically, you know, if I'm just to keep this simple because they actually do a lot of different things. But yeah, the primary thing they do is they're trying to fill a transportation need in their community and they are making car ownership accessible to those who would not otherwise be able to go out and purchase a car for economic, financial purposes.
And so, you know, they deliver a certain number of cars per year and it's really great, it's a great program. When you, when, when a recipient of a car buys a car and they do buy it, they buy it at an like, you know, $1,000 or less or something like this.
And it's, it's transformational for their lives.
Certainly the theory is that it's transformational.
It's really easy for them to measure how many cars they get out in a year. Really easy. That's, you know, and and that's the thing, right? Hey, we, we get cars out to people who need them, and we got 50 cars out this year. Wonderful.
But that's not really their end goal. Their end goal is that the lives of these people, because they now have ownership of their own reliable transportation, other things change.
And you, not too long ago did a sort of an evaluation report for them to help them look at what they're measuring, what they could measure, what they're finding so far. Car.
And the reason I bring it up is because of another element of outcome measurement, which is it's really easy to measure the stuff we're directly responsible for how many cars we get out.
[00:31:18] Speaker A: Right.
[00:31:18] Speaker B: It's a lot harder to measure what's changing for that car owner over the course of the next year to two years. Life of the car, beyond the life of the car.
And you know, there was, there was a trainer with United Way years ago, back in the early to mid-90s, who said that unfortunately, people are not like the elephants on National Geographic where we can tag them with this electronic tag on their ear and track them for 50 years.
And yet. Yeah, they kind of are. We kind of can, we kind of can do that. Right, so talk about this, talk about how. Because I think for a lot of organizations the difficulty is, man, you know, sure, we think that by getting them a car, we're improving their employability and their financial situations and their households, but we don't really know because we don't know how to measure that over time, track these people.
What would you say to organizations?
[00:32:17] Speaker A: Yeah, I would love to talk about that engagement because that was an engagement, for one thing. Patrick, It's a wild coincidence that we were both, out of the hundreds of thousands of nonprofits United States, that we both were involved with Second Chances Garage in different capacities. My social science consultants as the evaluator.
Yeah, and that was a very interesting engagement because first we worked with them on developing the theory of change.
Because at first if you ask them what they did, they said, well, we give away cars and like what? But you know, you have to press an organization.
And so with their board and some staff members, we came to.
Came to the conclusion that what they do is transform lives.
It didn't take that much prodding to get there. But then the question was, well, how do you transform lives? And who were you selecting? Whose lives are you transforming? Because it's not that they go out and have these incredibly low cost cars provided to low income people. They carefully select who will be the recipient of the Cars because they can't serve very many people at a time. They simply can't. It's one garage. It's one team of mechanics.
What they do is they get donated cars.
They will figure out what to do with those donated cars, whether they are used basically for junk value, whether they are good cars and they're sold to the public. And when they're sold to the public, then those funds support second chances garages, other aspects of the organization.
And the third thing is that they will rehab the car, make sure it's drivable at least for a few years, and they will sell it to an individual who is carefully selected for the program.
And the individuals, it's really interesting how they do it, because the individual, I think the price is about $1,000. The individual has to put up about $500 themselves because often there's another organization who will put up the rest of the money. So they have to have that money saved. They have to have, you know, basically some skin in the game.
So then the person will get the car. The car doesn't necessarily, in the first year of ownership, they don't actually own the car. I don't know if you realize this.
The individual doesn't own the car. There's a lien on the car. Because second chances garage learned that if you just give people cars for a low cost, they might turn around and sell them. Right. And so they want to avoid that possibility.
So they will keep a lean on the car for one year.
And if all goes well, after a year, the person gets full ownership of the vehicle, which is a great model.
And, you know, what we saw from our survey is that people, they did all sorts of things with their cars.
But, you know, important things that they did is that they increased the number of hours that they worked by about, you know, on average, eight hours a week, which, you know, that's a lot more income for the individual. They participated more in their community. They were able to get to parks. They were able to get their kids to educational activities. They were able to get their parent. People were able to get their parents to their medical appointments, Parents and kids and themselves to medical appointments. But it was. Was. The cars are absolutely transformational. I happened to speak with one person who I was speaking to because the person was in one of the partner organizations, staff of a partner organization. She had earned a master's degree, and she said that the reason why she was able to get her master's degree was because she had been the recipient of a second chances garage car at one point. And that changed her life. And so she was able to take advantage of nighttime educational opportunities.
So the organization then is able to speak about itself and has the data to support it in much broader terms.
I think more impactful terms. It's not that they give away cars, they do transform lives. They increase, increase the numbers of hours worked. And that, you know, for an organization that's huge because if you use that information for your fundraising, that will help you that, that will certainly be more attractive to donors than simply saying that we give away cars. So I'm a big fan of the organization, I'm a fan of the model, but I have to say it's not an easily.
It's not a model that can be replicated very easily because there are some distinct personalities, talents and the investment that an organization would need to make in just running the garage is pretty significant.
So any organization that thinks that they can just develop this car program, think twice. It's a big investment in second chances garage. They've been around for over 20 years.
They have figured it out. But it is, I think they do tremendous work.
[00:37:46] Speaker B: They really do. And I'm a big fan as well. And I'm glad you said all of that and described the program. And they're actually in growth mode right now.
They're out of room, they're landlocked and they're looking to expand their facility.
They can get more cars, they can get more cars out, they can sell more cars to help their revenue. So it's an exciting time for them right now. But yes, and what you're talking about I think is a good example of the difference between quantitative and qualitative data because I also, as I'm working with them on their next three years strategically. So the strategic planning piece of what they're doing and as a part of that, I'm interviewing a number of stakeholders and talked to a number of recipients of these cars.
And that's qualitative analysis. Talk to them, ask them questions. What did this do for you? Why was it different? Different?
You know, how did this, how did this expand out access for you to things like school and work? And you know, I found a lot of them are part time employed, but. Yes, but the car allows them to become full time employed as you said. So, so anyway, that, that's my point though is that it does take capacity for an organization to do that. That I mean you have to have some kind of mechanism rather than just, well, every once in a while, you know, we just hear our car recipients tell Us, these things, and we just kind of know them in the back of our heads. It's, you know, what we call anecdotal or whatever. There is a process and a structure to collecting data intentionally and over time and enough of it, that actually gives you meaningful work.
What would you say, what do you say to small nonprofits who say, you know, Beth Patrick, yes, I'm all in. This sounds wonderful, but you have no idea how limited we are in our capacity. We don't have time or staff to spend time talking to those people, collecting data, putting it somewhere, doing all this analysis. We just don't have the capacity for it.
[00:39:57] Speaker A: No. And they don't have the capacity, nor should they. Right. Because they're spending their time wisely in serving people or serving partners or, you know, in, in doing. In implementing their initiatives. They shouldn't have an evaluation capacity.
They're. They're too small for that.
So, yes, they should hire it out.
And I have found that funders, foundations, you know, some donors are very interested in evaluation.
They want to be sure that in the future their support for the program is money well spent.
So funders are often.
Are often eager to support evaluation.
[00:40:45] Speaker B: I want to pause right there, because you just hit on a good point. I think that's changed relatively recently.
[00:40:52] Speaker A: Do you think that's changed?
[00:40:53] Speaker B: I think over maybe, maybe a couple of decades, maybe even less, nonprofits have found it difficult to get money from foundations for anything other than direct service kind of work. You know, things like operating expenses and things like. That's always been a challenge for nonprofits to get from philanthropic funders and, you know, things like strategic planning, leadership development, outcome measurement, resource development.
I think it's been. I think it has increased over the last one to two decades of foundations who are themselves becoming more sophisticated and understanding about why they need to fund these things and not just insist that everything they fund is in the, in the form of some direct service or impact. I. That's been. My experience is that that has increased. It hasn't always been the case. Case.
[00:41:56] Speaker A: You're saying that they're now more eager to fund evaluation and strategic planning? Yes, I think that's right. I think that's right. Because after decades of funding initiatives and I, I mean, I speak with, with, with people who work at foundations, and I've heard some say, you know, we've been at this for decades, and we aren't seeing changes in our community, so we need to switch it up. And we. And, and so having evaluation is part of that switching it up. We need to take a really good look at what we're funding, what we're doing, what these organizations are doing. You know, there's also this big impetus. I mean, the Second Chances Garage evaluation that was supported by an independent foundation, by a family foundation. They gave Second Chances Garage money to go out and do an evaluation. And we were, fortunately, that the organization that they chose for that work.
But. Yeah, but, but, but I think that the foundations and funders want to see a change.
You know, there's a lot of talk about let's consolidate the nonprofit, all. All these nonprofit organizations as much as possible.
They want to know what the efficiencies are, what, what, you know, how can we be more efficient?
And that's also part of evaluation.
[00:43:16] Speaker B: Yeah, they don't want to double fund things. They don't want duplication. Exactly what I found.
Interested in your take on this, too, Beth?
So my dissertation was on nonprofit mission measurement.
And the question was, how are nonprofits measuring their performance against their stated missions?
So if my mission is change the world.
Right. How am I measuring? Do I have any indication, any evidence that I'm at least advancing my mission to fit, not accomplishing it? And when I went into my study, what I assumed I was going to find was that it was all about funding and capacity. That that's, by the way, that the answer was most nonprofits don't do well at measuring their work against their state admissions.
But the reason surprised me, and it shouldn't have. The reason, both in the literature and mostly in my own qualitative study that I did did was framing, not funding.
It wasn't that nonprofits don't want to fund the work. That's what I thought I was going to hear from everybody.
What it actually ended up being more often than not was if I'm not measuring against my stated mission, more often than not, it was because I had a poorly stated mission. Yes, the framing was wrong. And I think, you know, so you take that down into the program level.
What is the framing? You mentioned the term theory of change. We call it logic model. You know, the. It sounds like this big academic term. Right. Theory of change. All it is is we believe that if we do this, then this will happen.
It's an if then statement. Right, but we don't. Many organizations just don't. They've never formed that. They've never formed the actual framing of their work. Then measuring it actually becomes easier. Easier.
[00:45:11] Speaker A: Yes. Yeah. No, no, I totally agree with you on that. I'm not surprised that missions are so Broad that they become unmeasurable.
Because there was, I think foundations and funders have wisened up to this quite a bit. But, you know, so dial back 15 years ago, 20 years ago, there was this thinking that if an organization says that they will do bigger things, things that they have a higher probability of getting grants that they have applied for. Right. And so then what they've done is they've set the bar so high or so ambiguously that first they, they can't meet it.
So if it's ambiguous, then. Then you can't measure it. It's like, you know, what. Exactly.
[00:46:01] Speaker B: World peace promised here.
[00:46:02] Speaker A: Yeah.
And, and, and if the bar is high, then that gets them in trouble later on. Right. If you say that we are going to, to change, I don't know, increase reading scores by 20%, you know, that's an incredibly high bar. And if you increase them by 5%, that's great. Right. But it's not the 20% that you promised.
[00:46:22] Speaker B: Yeah.
[00:46:23] Speaker A: So there, there's this game that was played, and I think people are, are beginning to wisen up to it to some extent. I still think that there's, you know, that funders are partially at fault because they do want to fund the bigger, sexier thing. And so some. And you know, my love is small programs. I love small programs that you can wrap your, you know, you can wrap your head around it. You know what they're doing. Second Chances Garage is a perfect example of a small program that, that has achievable goals.
Right. You can measure it. They aren't trying to change the world. They're just trying to change the lives to some extent, of the people who they serve, who they carefully pick out
[00:47:10] Speaker B: for this program of Frederick County, Maryland. They're very. Frederick County, Maryland. That's right.
So you bring me to another question. And my gosh, I can't believe we're almost at an hour already. I'll need to wrap this fairly soon, but we could go on, I think, forever.
Let's talk for just a second. I have, I think, two more things I want to touch on.
One is attribution.
So, you know, yes, the. Let's take 10 recipients of second chances cars, and let's say that seven of them went on to make more money, you know, pay off their house, whatever, any kind of good financial things.
The question then becomes, how do you know that's just because of the car? Car.
Right. There's. There's a million other factors that come into the lives of these people.
And so attribution Often becomes a question, especially when it's some big collective impact initiative that a community is working on together, like housing and homelessness or poverty or any. It's like, how do you make attribution to the outcomes? And I, I want your take on this. I'll give you my perspective at least. What it's been, is that there's a reason we call it a logic model and not a proof model model.
You know, that logically there is enough data, if you do the literature review that tells us there are correlations between transportation access and, you know, a bunch of other indicators.
So the logic model says, well, okay, if that's true, then logically, if we can get more cars into the hands of people that otherwise wouldn't be able to get them, we can, we can increase the amount of, of those, of those outcomes.
So it's not a proof model and it's, it's a logic model. It's a theory of change.
[00:49:08] Speaker A: Right.
[00:49:08] Speaker B: Not a law of change.
[00:49:09] Speaker A: Right, right, right. Yeah. We aren't in court.
Right. We're just trying to build the argument.
[00:49:17] Speaker B: Funders want that. I mean, do they at least the.
[00:49:21] Speaker A: I don't think so. I think that funders are thrilled if you can show through a rigorous, by contracting with an evaluator, having a professional look at your, a disinterested professional look at your programs and give you honest feedback on them, do a rigorous evaluation, sketch out the theory of change and the program logic model and do the literature review, conduct the interviews, conduct the surveys, look at all the data that has been collected and tell the story.
And then, you know, any evaluation should also have recommendations, recommendations for improvement, for expansion, for, you know, changes that need to be made, whatever that.
Yeah. So no, we aren't trying to have a, you know, we aren't in a court of law.
So yeah, just build the case.
[00:50:27] Speaker B: Yeah, that's right.
Last question. I think I'll unpack with you regarding this stuff is what about AI?
You know, I mean, even, even decades ago when I was doing some outcome measurement in the United Way world, there were soft. You know, there are software programs that help you collect data, track data and communicate outcomes and things. But I'm going to tell you, AI as an extens. User of agentic AI today I, I can tell you that qualitative data coding, for example, the way that, the way I had to do it for my dissertation using, you know, NVivo software and do my own theming and coding and all of that, AI will do that today. AI will do for organizations that are struggling with. Man, I don't know how to use. I don't know how to code qualitative data. I don't know how to know how to. There, there's, you know, you obviously have to be careful with it, just like you have to be careful with it in any other application.
But I'm interested in any of your experience with AI when it comes to program evaluation. Anything from data collection to analyzing the data, predictive data. Now coming out, that saying, if you do this, this, you know, chances of that.
Give me your thoughts on AI with regard to evaluating our work.
[00:51:47] Speaker A: Gosh, I'm so old school. I really am. I've, I, you know, just out of curiosity, I'm in the middle of an evaluation and I put the interview. No, it wasn't even the interview. I, I put the notes that I had on all of the meetings of a Collective Impact Initiative into AI just to see what it would give me. And it gave me garbage.
It gave me just this generic language and then I went back into the interview notes and then I prompted it and I asked for specifics and it gave me specifics, but they were specifics for the wrong areas and they were specifics that didn't necessarily exist to the extent that AI would have of that AI implied.
So I am shying away from AI for any sort of qualitative analysis. So I'm very old school. I just don't think that it.
I, I think that having a, having AI look at qualitative notes, and my notes are extensive when I, When I conduct interviews, when I take notes on meetings, anything like that, I think that relying on AI is not relying on my experience.
And I think my experience is very, very valuable.
[00:53:18] Speaker B: No doubt about it.
[00:53:20] Speaker A: I have bright white hair and that white hair has been earned and it is valuable experience. I've worked with many, many, many organizations and having AI go into it with basically a blind eye and come up with sometimes gibberish, I think if I relied on that, that would be a real disservice to clients and to my field.
So, yeah, I'm not there with it.
[00:53:50] Speaker B: I hear you. No, that's good. And I don't disagree with anything you just said here. Here would be a couple of questions that I'm curious about.
When you put in the stuff on the Collective Impact stuff and you got back junk, how long ago is that?
[00:54:07] Speaker A: It was last week.
[00:54:09] Speaker B: Okay.
And what tool were you using?
[00:54:13] Speaker A: I was using 365.
Microsoft.
[00:54:18] Speaker B: Copilot.
[00:54:19] Speaker A: Yeah, copilot. Sorry. Yeah.
[00:54:21] Speaker B: So here's What I've learned. And again, as a really extensive user of AI, I have the opposite. Opposite. I have the opposite experience. But, but it's because of some of the learning and the training that I've done on the AI stuff. Here's what I've learned. Here's what I'm thinking about AI.
There was a meme not long ago that showed that. You remember that. Remember when Microsoft Office was first. The window, the first Microsoft Windows Word, and the little clip, the little paperclip animated paperclip guy called Clippy that would come on and say, hey, it looks like you're writing a letter.
Well, that was, that was AI a long ago, sure, yeah, but it was AI and it was, you know, the, the statistical plugins now in Excel where you can do T tests and Z tests and regression analysis and all that without knowing statistics. That's AI.
All of those are. And the, the, the key is that we've evolved to the place where you still have to know how to use those tools.
And what I've learned about AI is one, there is a huge difference between opening them up, opening up Chat GPT or Copilot or any of these things and free versioning it and using the chat bot and throwing some stuff in it and say, hey, tell me about this. There's a huge difference between that and sort of programming and training an AI agent.
And I spend money on AI, so I don't use the free versions that can do much, much more.
But the key is in learning how to prompt.
[00:56:07] Speaker A: Sure.
[00:56:08] Speaker B: Big, big difference.
I've watched organizations in real time. Oh, let me go to ChatGPT and ask it. No, no, no, no, no, no.
That's, that's like Siri don't do that. You have to learn how to prompt it.
And the prompts of. I mean, there are prompts that get rid of the AI hallucination that get rid of, you know, some of these other language things.
So for example, I did a comparison. I coded some Data in using NVivo, which is a tool that a lot of academics use, and coded the data as I was taught to code it in school and think.
And then I put the same data into Claude Anthropic's version and using Claude Cowork or Claude code, not just the Claude Chatbot.
And what in the prompting. And I won't go through the whole prompt, but it was about use.
I gave it the rules and guidelines and told it where to go and research how to theme and code data the way in vivo or any other Qualitative software would do it. Give me a code book for it.
Show me, like in school, show me your work.
And it came back almost identical with the way I had themed it and actually much better in some ways. As I looked at some of what it did, I thought, oh, I missed that.
That should have been coded this way.
So I would say, I think we're probably, probably both Right. Beth, if I'm being honest, if you just. You. Your words were if I rely on AI, I, I agree. I don't think we rely on it, but I think it can be a useful tool to supplement what we're doing if we know how to use it and if we are able to understand whether or not what it's getting back for us is what we're really looking for.
[00:58:06] Speaker A: Right, right. And. And that's where you need the background and you need to be able to ask the right questions.
[00:58:15] Speaker B: That's right.
[00:58:16] Speaker A: So having AI, like, if people think that they're going to be able to conduct a good evaluation with, you know, relying totally on AI, I think they will never get a good evaluation out of it. The dangerous thing is, is that they will get some sort of evaluation out of it and then they'll think that, well, we did our eval evaluation.
[00:58:40] Speaker B: You cannot blindly trust.
[00:58:41] Speaker A: And then they'll trust it.
[00:58:42] Speaker B: Yeah, you can't blindly trust it. That's exactly right. I mean, I've. I've definitely gotten back some of the junk that you're talking about.
[00:58:50] Speaker A: Yeah.
[00:58:51] Speaker B: It's not perfected, but the reason I ask, when you did it. And it's interesting that you did it a week ago. Yeah, this. The models are changing so fast.
[00:59:01] Speaker A: Right.
[00:59:02] Speaker B: They're becoming so much more sophisticated. And I mean, I mean, doctors are using it now as their scribes in the healthcare system, things like that. So anyway, good. That.
[00:59:14] Speaker A: No, they're wonderful uses for it. And I think in statistics, I think that there's some very, you know, in big data analysis, there are wonderful uses for it.
[00:59:25] Speaker B: Yeah, I hear your caution, though, but
[00:59:27] Speaker A: I think with evaluation.
Yeah, I think you need to be very careful because evaluation is a social process, it's a political process, and, you know, in addition to a straight data analysis process. So. Yeah. So you need to be very careful. Yeah.
[00:59:47] Speaker B: Boy. Well, Beth, again, we could go. I am not out of questions and topics on this thing, but we will wrap this up. I thank you. Your insights are, are so helpful, so valuable. I do want to point people back to socialscienceconsultants.com and if you Are an organization out there looking to get some help on this because you know you want to do it, but you don't have the internal capacity for it.
This is why you would reach out to a Dr. Beth DiPonte. So socialscienceconsultants.com and Beth, there are two questions I like to ask all my guests before I wrap up a show because I love the answers. And this isn't about outcome measurement.
So I'm going to get you out of that, get out of that gear for a second. First question is, who comes to mind for you as a leader in your life? Whether it's someone you know or just someone you've observed from afar, or even someone from the past, who comes to mind as a leader that has had what you would call profound impact on your view of leadership and why.
[01:00:54] Speaker A: So the person who immediately comes to mind is Joyce Rothermel. And Joyce was one of the founders or the founder of the Greater Pittsburgh Community Food Bank.
Joyce is a former nun and she left being a nun. And I would see Joyce in action when I was a young academic interested in food assistance.
I would see Joyce go into a meeting and very nicely say to everybody, oh, did anybody bring an agenda? And everybody else would say no. And then she would hand out her agenda, which that takes a lot. And so her leadership is just incredible. And the Greater Pittsburgh Community Food bank is one of the country's leading anti hunger organizations, national now. And it really is because of Joyce's work. She was a huge contributor. So she could rob funders blind. She could talk to them.
She was, she was just amazing. She's retired now and I hope in good health. And. Yeah, she was just wonderful.
[01:02:04] Speaker B: What a great tribute. What a great tribute. And what an impact that. Yeah, something like that. Interesting how you remember the, that with that one little mechanism. Right. Does anybody have an agenda? And. Well, I came, I came prepared. But the idea that she opened it up to others first and yet. Absolutely. And yet still came prepared. That's beautiful. That's awesome.
[01:02:24] Speaker A: Yeah, yeah.
[01:02:25] Speaker B: Last question, Beth. And, and then we'll get on about our work. But you're at the top of a mountain and you got a, a big megaphone and all the leaders of the world are at the base of the mountain to HEAR what Beth DiPonte has to say about leadership. And you've got 15 seconds to tell all the leaders of the world what you believe is the most important thing for them to keep in mind as they lead. People, people, what is that? Beth Duponte, 15 second soundbite on leadership.
[01:02:56] Speaker A: It would be admit your mistakes, correct your mistakes and be kind.
You know, just be kind to each other.
Yeah, that would be it. Pay your staff as much as the as much as you can.
Pay it forward and be kind.
[01:03:13] Speaker B: I love it. Admit your mistakes, correct your mistakes and just be kind. Those are easy things, Beth. I think we could all do that if we wanted to.
[01:03:22] Speaker A: I hope so. I hope so.
[01:03:24] Speaker B: Beth, thank you so much. Boy, maybe one day we do a version 2 or a 201, a continuation of this conversation because this has been great. Thank you for your work.
[01:03:35] Speaker A: Thank you very much for giving me the opportunity to speak with you.
[01:03:39] Speaker B: Yeah. And folks, if you want to dig deeper into the work, pick up a copy of Evaluation Essentials. It's a Wiley book. And visit socialscienceconsultants.com and until then, we'll see you next time. Lead on.
