TORONTO (Reuters) - For Canada's banks, an increasingly gloomy economic picture combined with regulatory uncertainty may soon sap the momentum they managed to sustain through their latest quarter.

The banks reported a decline in capital-market trading income for their recently completed financial third quarter that wasn't worse than expected, which spurred investors to breathe a collective sigh of relief.

But, overall, Canada's "Big Six" reported a decidedly mixed performance. Two banks topped analysts' estimates, two missed, and two were roughly in line.

Even so, the Toronto Stock Exchange's financials sector rose 4.2 percent over the past two weeks as the results trickled in, outperforming the 3.6 percent rise of the market's main index.

Observers say things won't soon get much better than this as growth in core retail banking starts to stall. Business and personal loans, particularly mortgages, are expected to flag along with an expected slowdown in economic growth.

"I think, going forward, the issue becomes: Do you get some kind of bounce back in capital markets, and, if so, is that going to be offset by weaker trends in the Canadian housing market?" said Sumit Malhotra, an analyst at Macquarie Equities Research.

Canada's housing market suffered a much more mild slowdown than the US market, and its quick rebound helped bank profits recover quickly from the financial crisis.

But, even though Canadian interest rates remain low, housing sales have begun to cool, prompting Toronto-Dominion Bank (TD), which closed out bank earnings Thursday, to warn that rapid growth in retail bank revenue could soon end.

TD, one of three Canadian banks active in the United States - Royal Bank of Canada and Bank of Montreal are the others - faces additional headwinds from the even gloomier economic growth picture south of the border.

"Those were pretty good numbers (during the quarter), but the warning sign is that in the US you're not seeing anything like a healthier economy, you're not seeing consumer spending growth," said Gavin Graham, president of Graham Investment Strategy in Toronto.

With uncertainty surrounding the housing market, valuations suggest bank stocks may have trouble making gains unless another catalyst emerges, analysts say.

"When you look at where they're trading on a relative to historical basis and on a forward price-to-earnings basis, they're actually reasonably valued," said Barclays Capital analyst John Aiken.

Current price-to-earnings ratios for the banks, based on next year's expected earnings, are in the 10 to 13 range, which analysts say is normal for the sector.

Aiken notes the Canadian banks are trading at much higher relative valuations than their US counterparts, which have yet to fully rebound from the financial crisis.

"If you're looking at an economic recovery scenario, particularly in the US, the Canadian banks don't have that much multiple expansion left in them...So anyone who's optimistic on the (economic outlook) is kicking the tires on US and European banks, not Canadian banks," he said.

A resumption in dividend hikes could spur share prices, but banks are waiting to act on dividends until the Basel global banking committee comes up with tighter capital and liquidity rules, which will be aimed at avoiding a repeat of the financial crisis.

While the banks will likely get the green light to raise payouts when the Basel rules are released in November, only one or two lenders are expected to respond immediately with hikes.

Analysts expect the banks to try to maintain dividend payout ratios - the percentage of profits committed to pay the dividend - in the 40 to 50 percent range, meaning banks that are at the high end of that range would likely wait until profit increases before they would hike the payout.

National Bank of Canada, Canada's No.6 bank, currently has a ratio around 40 percent and the bank's chief executive, Louis Vachon, said last week that National would be "a lot more precise about our capital plans" when it holds its next quarterly conference call, a statement some took to mean a dividend hike is coming.

TD, which has a payout ratio around 47 percent, is expected to raise its dividend early next year.

The remaining Big Six, whose ratios are around or above 50 percent, will likely wait a few more quarters, which would delay the valuation bump that would come with the move.

"I think that near term, some of the chatter surrounding dividends could give the banks some support. But in our mind that needs to be joined by a re-acceleration in revenue growth to significantly get the stocks moving," Malhotra said.