Former Tottenham boss Sherwood, 48, launched the tirade in the tunnel at half-time of February's 1-0 Sky Bet League One loss after Brown had awarded Bury a penalty for a shirt pull by Raphael Branco.
The FA on Monday published the written reasons for Sherwood's suspension, and Â£2,000 fine, and included Brown's report on the incident.
Brown wrote: "After the half-time whistle, and as I was walking down the tunnel, Tim Sherwood of Swindon Town approached me and said 'You're the only f***** in the ground that's f****** seen it you f****** mug'
He then started pointing aggressively in my face.
"I told him to calm down and stop pointing at me or he would be sitting in the stands for the second half
He then continued 'I wasn't even in the dugout you f****** mug and you would be doing me a favour'.
"I told him I wanted to speak to him at my dressing room but he ignored me and walked off towards the Swindon changing room
I followed him and stood outside and told the coaching staff I wanted Mr Sherwood to come outside as I needed to speak to him.
"After approximately 45 seconds Mr Sherwood walked out and walked past me and I informed him that he would be reported for his comments towards me.
"His reply was 'f*** off you mug'
I then entered my dressing room area and informed the stewards of my decision and that if Mr Sherwood tried to come down into the technical area he should not be allowed."
Sherwood has served his ban, with struggling Swindon winning both matches in his absence.
Source : PA