A few years ago, when I was teaching a master’s degree course on data analysis for HR, we had a session on the role of data in performance appraisals and coaching. Our discussion started by looking at the effective use of data in the appraisal process, but then we turned to look at measuring things after the fact. An interesting question that came up was, “how do you know that coaching works?” The coaching we were talking about was not the informal coaching that might happen between a leader and an employee, but instead, assigning a formal coach who may be a certified individual from within the organization or, more likely, is a professional coach from the outside. A lot of time and money are spent on coaching, yet we often do not follow that up with a solid evaluation of whether or not it helped, once the individuals performance has been assessed.

There is a reason for that, of course: it’s tough to do. It is easier to simply assume coaching works (because any concept that has created such a large industry must work, right?) than it is to take the time to figure out if it really is helping.

In general, I am a fan of coaching (and no, I do not currently work as a coach myself). I think it is helpful to have an outsider provide advice and feedback on specific leadership issues without getting caught up in office politics and other distractions. Coaches also provide good general guidance for people who are new to leadership or experiencing radical change. At the same time, I think the proper role for coaching depends on the needs of the firm and the leader, so blindly adopting a generic approach may be about as useful as flipping a coin to make your decisions.

The main problem with evaluating coaching’s effectiveness is twofold. First, it is difficult to really know how well a leader would have done without coaching. You can try to identify specific decisions they made as a result of coaching, but would they have made those same decisions anyway? It is possible to correlate the topics discussed in coaching sessions with the decisions a leader ultimately makes, but that takes time as well as a good knowledge of everything the coach and the executive have discussed.

Another problem is deciding what to measure and knowing how much impact coaching had on it. From a firm’s perspective one of the ultimate measurements is profit (obviously, this is different in the public sector, but every organization can define some primary goal). So, if profits are steady for a while, then coaching begins and soon after that profits start to rise, well, it must be due to the coaching, right? Well, maybe. Or maybe the economy picked up. Or you have hit a seasonal high. Or…well, something else. You need to isolate the coaching from other factors, and again, this is going to take some time and effort. Not many firms seem to want to do this.

One question, of course, is who should do this evaluation? The coach has an economic incentive to show that coaching works. The boss of the person getting coached could evaluate the coachee’s performance (I don’t think “coachee” is really a word, so I’m making it up now…feel free to use it!) but that boss will not know the content of the coaching sessions and would not have evaluated the coach’s style. In the end, the best person to evaluate the effectiveness of coaching may simply be the person getting coached. It is not totally objective, it is not exactly unbiased, but it may be the best you can do.

Coaching can be great, and even if it is tough to measure its effectiveness, maybe you should do it anyway. Maybe. The best way to approach it is to understand WHY you want to do it, and have some goals for the coaching program. Then design a plan for meeting those goals, and ask yourself later if you met them. It may not be perfect, but it might be better than what you’re currently doing.