A study of the reaction e(+)e(-)->mu(+)mu(-)gamma(ISR) at LEP and search for new physics at annihilation energies near 80 GeV