In the Alerts view you'll see all existing alerts and an option to create a new one.
To setup a new alert:
Select the query you want to trigger an alert for (just start typing) - queries with parameters won't work
Select the column you want to watch
Select the trigger type (greater/less than or equals to)
Select your magic number
Leave Rearm Seconds empty to get 1 alert when the status changes from Triggered to OK, enter a number to get an alert every time the query runs (by schedule) + the Rearm seconds value.
Define alert destinations - email, Slack, HipChat and webhooks are supported.