The stability criterion of a regular vortex pentagon outside a circle

The nonlinear stability analysis of a stationary rotation of a system of five identical point vortices lying uniform on a circle of radius $R_0$ outside a circular domain of radius $R$ is performed. The problem is reduced to the problem of equilibrium of Hamiltonian system with cyclic variable. The stability of stationary motion is interpreted as Routh stability. The conditions of stability, formal stability and instability are obtained subject to the parameter $q=R^2/R_0^2$.