I went to an awesome talk on creating high performance Drupal sites by Josh Koenig of Chapter Three at the recent BAD Camp in Berkeley. You should check out the slides, but here's a recap:

1. basic performance settings at [domain]/admin/settings/performance (compress css, page caching, etc.)
2. use Drupal Pressflow (optimized version of Drupal): https://launchpad.net/pressflow - you can just drop it in as a replacement of the Drupal core which supposedly no side-effects (except higher performance)
3. add in "Varnish" to stack above Apache - will return cached versions of files for anonymous users - faster than something like boost.module because it never even gets to apache - http://en.wikipedia.org/wiki/Varnish_(software) http://varnish.projects.linpro.no/
4. configure php opcode caching such as APC - will cache all of drupal code (if cache size is configured big enough) so that the code doesn't need to be re-interpreted every time - http://php.net/manual/en/book.apc.php
5. use solr for high performance search instead of Drupal search - http://lucene.apache.org/solr/
6. use memcached or similar for database cache - http://www.mysql.com/products/enterprise/memcached.html

Implementing Varnish alone supposedly makes an *extremely* big difference (e.g. 200 times faster to load a page with Varnish than through Drupal).

So, if you want to have a super high performance Drupal website, follow Josh's advice and you'll be in great shape. Thanks to Josh and all the BADCampers for the great conference!