Not only the idler gear bearing fit and shimming is important (0,4-0,6mm -.015 to .025in) along with the proper gear play. The gear play should be set so there is an audible click all the way around, not less as the basket expands when hot. That is adjusted with shims at the gearbox lugs.
But also what is important is the clutch basket bushing for the mainshaft. The basket is only supported on a single row ball bearing and the basket can rock on the shaft if the bushing is bad, inducing oscillation! Specially bad if the basket is a bit oval, as it tends to be in most engines, more or less.
You can check furter with magnet fishing at the bottom of the gearcase, if you catch a lot of metal debri it might be a sign that the gears doesn't have the right distance from each other. I would make a steel wire hook, and try to lift and rock the countershaft to check if the clustergear or the shaft is loose in its bushings, the one that wear most is that bushing close to the sprocket. The shaft should be firm and lightly knocked in the sprocket side bushing and push fit in the clutch side bushing.
The wall in the gearbox for the countershaft bushing on the sprocket side isn't that strong, it absolutly needs support on the outside when a new bushing is knocked or pressed in. Heat the case before, but not more than spit sizzles on the area. A sensible modification on the cluster gear shaft is to drill and thread the clutch side end for a retractor bolt..