See Gotō Islands#Talk.
Except for the few interesting sights inside Fukue City, you will need a car to get around. Buses are few and far between (consider the tourist buses below), and taxis would cost you: a roundtrip to the Osezaki lighthouse and back would be about 20.000 yen.
Unless you´re very confident cycling, you won´t be able to move around the island with a bicycle: Fukue is too big and too hilly for that.
Inside Fukue Town:
- Castle remains: the feudal lord of Goto waited for around a century for the authorization to build this castle, and when if finally came, only 4 years after completion the Meiji Government emitted the Castle Abolition Order (1873). Today only the walls and pit remain, but it's well worth a visit. Now the Goto High School stands inside the walls.
- Old houses' street (buke yashiki dōri): 2-3 min. walk from the castle you can find a street that retains the stone walls and doors from the houses of the noble families that once lived here. Regular modern houses stand inside, but there's one house with an old look. They sell souvenirs and rent bicycles (500 yen half a day).
- Myōjindo: a small temple constructed for protection in the sea by a Chinese merchant that moved to Fukue. Off the commercial street and next to the river.
Around the island (clockwise starting from Fukue Town):
- Onidake: the grass mount that dominates Fukue City is a good picnic spot. They also have cardboards so you can enjoy "grass skying".
- Camelia park (Tsubaki-en): just next to Onidake, the name says it all.
- Abunze lava coast: a coastal lava formation.
- Osezaki lighthouse: by far the most famous sight in Fukue, it has been featured in the film Akunin.
- Takahama beach: a popular spot for swimming in summer.
- Dōzaki Church: a church turned into a museum about Christianity in Goto.
If you find yourself car-less in Fukue, probably your best option is to book a tour with the bus tour company at the ferry terminal. Between the morning and evening tour they cover the main sights of the island. Even if you don´t understand Japanese, they are still the cheapest way to get around.
If you do speak Japanese, phone before hand to reserve (see bottom).
Course A (morning):
2100 yen (plus 300 yen entrance fee)
Starts at 8:50 Fukue Airport, 9:20 Fukue ferry terminal
Covers the Dozaki church, Abunze lava rocks, Onidake (no stop) and Old houses´ street (no stop).
Finishes at 11:15 Fukue ferry terminal
Course B (afternoon):
Starts at 12:35 Fukue ferry terminal
Covers the Osezaki lighthouse, Takahama beach (possibly no stop), and Mizunoura church (no stop).
Finishes at 15:50 Fukue ferry terminal, 16:10 Fukue Airport
200 yen discount if you book both tours.
They usually require a minimum of two people to conduct the tour, except between July 19 and August 31. If you can speak even a little Japanese, try to phone before noon the day before to reserve and make sure they'll do the tour: 0959-72-2173.
For up-to-date info and details check their homepage (Japanese only). 
Yagurazushi and Torazushi, both in the commercial alley at Fukue City.
While even Fukue City can get very dark at night due to no lighting, Fukue is as safe or more as the rest of Japan. Just exercise basic caution and you´ll be fine.
You can hop to the other islands of the archipielago by ferry. See Gotō Islands#Get in.
Else you can go back to the Japanese mainland, either Nagasaki or Sasebo, by boat or plane.