Pension funds aren’t getting a good deal on Wall Street.

The states paying the most to outside money managers have produced lower investing profits than those that have paid the least, according to a new report.

Cash-strapped pension funds are increasingly questioning the worth of external money managers. As pension costs absorb a bigger part of government budgets, the sometimes lavish fees paid to hedge funds, private-equity firms and others is falling under harsher scrutiny.

“Ironically, higher Wall Street money-management fees lead to lower returns,” said Jeff Hooke, a co-author of the study and a senior fellow at the Maryland Public Policy Institute, a conservative-leaning group. He was involved in a 2012 analysis, which reached a similar conclusion that higher fees didn’t pay off.

Over the past five years, the 10 states cutting the biggest checks to external firms produced an average return each year of 12.44%, according to the Maryland Public Policy institute. That trails the 12.7% at the 10 lowest-paying states. The totals are net of fees.

The states paying the highest fees were Missouri, South Carolina and New Jersey. States paying the lowest fees were Tennessee, Oklahoma and Georgia.

The study analyzed annual filings of the largest pension fund in 33 states. The other states were omitted because their fiscal years ended on dates other than June 30, 2014.

The states collectively paid $5.4 billion in external-management fees in 2014, according to data compiled by the Maryland institute. That is a roughly 10% increase from 2012’s $4.9 billion, despite two more states being included in the total.

The rise can be attributed to a bigger helping of hedge-fund and private-equity investments by public pensions, which increase fees, Mr. Hooke said.

Write to Timothy W. Martin at timothy.martin@wsj.com