If you want to activate Guts, then give the pokemon a Status (Paralysis, Burn, etc) -- just like the game. If you want to know what the Ability Effect checkbox is for, then read the help page for it.

Doug, I was fiddling with the Smogon calculator for theorymon calcs and I encountered an error. When I wanted to test a Guts boosted Collosoil Earthquake I set ability to Guts and click the 'ability effect' checkbox to on, but nothing happened. The calculator output was the same for the ability effect checked and without. Just wanted to bring this to your attention so it could be fixed.