Was anything logged in your slow query log?
I have created a demo board with > 100 000 topics and 600 000 posts and it's loading almost immediately with 2 topic blocks, one fetching the last topics sorted by creation date and the second one with an additional where condition for few specific forums.
What happens when you run this query in phpMyAdmin
SELECT forums_topics.*, author.*, last_commenter.*, core_tags_cache.tag_cache_text FROM `forums_topics` LEFT JOIN `forums_forums` ON forums_topics.forum_id=forums_forums.id LEFT JOIN `core_tags_cache` ON tag_cache_key=MD5(CONCAT('forums;forums;',forums_topics.tid)) LEFT JOIN `core_members` AS `author` ON author.member_id = forums_topics.starter_id LEFT JOIN `core_members` AS `last_commenter` ON last_commenter.member_id = forums_topics.last_poster_id WHERE forums_forums.password IS NULL AND forums_forums.can_view_others=1 AND approved=1 AND topic_archive_status=0 AND forum_id IN (2,5,6) AND forums_forums.min_posts_view<=110715 AND forums_topics.approved < 2 AND forums_topics.approved!=-2 AND forums_topics.approved!=-3 AND ( NULLIF(forums_topics.moved_to, '') IS NULL ) ORDER BY start_date DESC LIMIT 6
replace forum_id IN (2,5,6) with the ids of the forums you want to query