Periodically Reassess JVM Optimization for Significant Java Applications
The various Java Virtual Machine (JVM) garbage collectors all have much the same performance at small scale and for simple applications, or at least the differences are going to be small enough that they can be ignored for any practical purposes. It isn't worth the effort to profile and optimize JVM settings for a casual application, and doing that work isn't even good practice for working with larger, high traffic Java applications. It is, however, well worth profiling larger applications under load, trying out the different garbage collectors, and toggling a few other related options in the latest JVM implementations. There are many good optimization guides out there, but make sure you are working from a recent one, appropriate to the JVM implementation and Java version in question.
The point I wished to make here is this: given the changes implemented over the past few years of Java 7 and 8 development, the JVM optimization for any large Java application should be periodically reassessed. This is especially true if it was first put together in the Java 6 era or earlier, when a more limited and less effective set of optimizations were available. There is a tendency for the team that maintains a high-traffic or otherwise significant Java application to go with the flow and not rock the boat, working under the assumptions that (a) the JVM settings established during earlier development and release cycles are still close to optimal, and (b) that changes in the codebase since then have not introduced significant alterations into the application load and garbage collection characteristics. Neither of these are particularly good assumptions, especially as the years add up. In fact for any large application, rerunning a process of profiling and exploration of JVM optimization options every year or so is probably a good idea.
It is also a good idea to export JVM metrics into a monitoring tool and make that a part of the application dashboard. This is easily accomplished with minimal effort when using third party services such as Datadog and the like, but can also be set up internally to an organization using open source tools like Zabbix, Nagios, and so forth. The first step along the road of letting your Java application drift away from optimal performance is to fail to pay attention to ongoing changes in JVM metrics, especially those involving garbage collection performance. If you don't have the metrics to hand, issues with garbage collection can manifest in ways that look a lot like thread starvation at higher server loads. Garbage collection issues are nowadays much more complex than just an obvious halt in processing across the board, and this is why one has to pay attention to the JVM internals. Developers can spend a lot of time chasing red herrings through a codebase and its thread pools rather than focusing on the culprit that would be obvious given a record of garbage collection metrics across recent history.
Java now has a number of different garbage collection implementations. For large applications the more recently developed Garbage First (G1) collector is well worth testing. For web applications with large heaps and thread pools it is typically a significant improvement over the default parallel collector; I've seen one large-scale application in which that change halved the number of servers required at the same time as it improved request response metrics. G1 may well become the default collector in later versions of Java on the strength of this sort of result. As noted above, for legacy applications that have migrated from Java 6 or earlier to Java 8 or later over the years, and which were thus originally launched prior to the availability of the G1 collector, it is especially important to reassess optimization - and giving the G1 collector a try is the first and most obvious step in that process.