What matters in evaluating charities is results, but most charities either don't have or don't share the necessary evidence. The GiveWell approach with AMF, SCI, and Give Directly has been a combination of in-depth research into the specific charities and looking at the non-charity-specific evidence for the intervention in question. Unfortunately this is very time-intensive, which means it's expensive on a per-charity-fully-evaluated basis. For identifying the very best charities this is fine, but if you're Charity Navigator and want to rate the whole field it just doesn't scale.

While I didn't see any way to fix this, I've now read their description of CN 3.0 (pdf), and I'm actually kind of optimistic. The idea is that you change the incentives so that charities are putting more information out there, and then you can evaluate them better:

CN 3.0 seeks to change this by recognizing charities that do a better job of reporting their results. This provides effective charities with a particular opportunity to demonstrate their effectiveness by reporting their results. As more charities report their results, givers and social investors will have more and better information to inform their decisions.

In the future, when more results information is available, Charity Navigator will assess those reported results. But before it is possible to evaluate results, it is necessary to shift the paradigm of charity reporting from the current norm of selectively reporting, such as storytelling or case studies that may not be representative of overall performance, to reporting on demonstrably important measures, and showing how the organization learns and improves based on those measures.

Here's an example of that results reporting:

more...

While this isn't perfect, it's still very good news: Charity Navigator is influential [1] and shifting money to more effective programs is valuable.

(The nonprofit reactions I can find are pretty negative: Charity Navigator 3.0: The Empirical Empire's Death Star? and Charity Navigator 3.0: Missing the (deeply human) point of philanthropy?)

[1] GiveWell may be catching up, though. In 2012 they moved only 20% less money to charities through their website than Charity Navigator did through theirs. If that's roughly proportional to total money moved then they're in the same range, but CN estimates their total annual money moved at $5 billion to $10 billion while GiveWell estimates only $10 million. That means Charity Navigator is estimating a 2500x multiplier while GiveWell is estimating only 4x. This is kind of strange. It doesn't look like GiveWell is pushing donation much harder than Charity Navigator is (screenshots: GiveWell, Charity Navigator) so I don't know why their multipliers are so far apart. In GiveWell's case you can get an upper bound on money moved by looking at total funds received by recommended charities, but that would be an extremely weak upper bound if applied to Charity Navigator. I don't know what's going on here.