Basically if it's below 50 degrees, have your ass puckered. If it's below 42-45, drive something else (or have all-seasons on it). Rain only matters if it's cold.
Realistically, since mine is going to be my only car, and work pays for public transport... I'll drive it to work above 45 degrees (even in rain), otherwise I'll ride the bus and leave the 1LE in the garage - Keeps the car cleaner longer too