A top official warned of widening loan fraud in 2004, but the agency focused its resources elsewhere.