PHP mail https php.ini secure speed question

VPS customer using php mail for user signup. The mail script executes in about 20 seconds now that I have https, it used to only take 1-3 seconds tops. Is there a setting I need in my php.ini to cause this to execute faster? It's really bad for ux and I've already troubleshot the rest of the script, if I omit the mail function, it takes about 2 seconds to execute. Here, it's all textbook, take a look, if you want. $to = "$e"; $from = "My site name "; $subject = 'Account Activation'; $message = 'my site'; $headers = 'MIME-Version: 1.0' . "\r\n"; $headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n"; $headers .= 'From: $from' . "\r\n"; mail($to, $subject, $message, $headers);