Some performance pointers:
Setting the file system to private may reduce performance as Drupal is fully bootstrapped on every file request when set to private.