Can I grow a poon tree in my yard?

I live in new england, and it gets a bit nippy at night. I really need to grow a poon tree, also known as any plant from the Calophyllum genus. I would also like to know how long it will be until my poon tree is usable.