Shadow Finance Minister James Patterson accused the Reserve Bank of Australia of delivering a "vote of no confidence" in the government's economic agenda Tuesday after the central bank lowered productivity growth expectations to what he called "crisis levels."