We almost missed our plane going to Bermuda and wished we had. We went for 5 nights and strongly considered leaving early.
To be fair, we went the first week in January. A friend of mine had gone 2 weeks before us and had beautiful weather. It was freezing cold when we were there. The entire time we were there, all of the locals kept telling us that they can't even remember when they had weather this cold. There was absolutely nothing to do here. You couldn't swim or do any outdoor activities. The beaches all looked closed. We ended up having to buy long sleeve shirts and there were not many to choose from.
A big negative is that you can not rent a car. You must rent a scooter, which we heard was dangerous and lots of people get in accidents when they are not used to driving on the left side of the road. OR your other option is to rely on public transportation (buses). Taxis were not readily available.
We ended up just touring Bermuda. We walked and walked and walked. There was one end that was more historical. We saw a church that was never completed and were told that they ran out of money so they stopped building it. It has been sitting there for years. We went to the other end of the island, where there were some pottery shops and ate at the Frog and Onion where we had burgers and ale. Everywhere we ate, the food was just average.
Even if the weather had been nice, this is not a place we would return.