In this paper, we study the short-time behavior of the at-the-money implied volatility for arithmetic Asian options with fixed strike price. The asset price is assumed to follow the Black-Scholes model with a general stochastic volatility process. Using techniques of the Malliavin calculus such as the anticipating Ito's formula we first compute the level of the implied volatility of the option when the maturity converges to zero. Then, we find and short maturity asymptotic formula for the skew of the implied volatility that depends on the roughness of the volatility model. We apply our general results to the SABR model and the rough Bergomi model, and provide some numerical simulations that confirm the accurateness of the asymptotic formula for the skew.