Low latency logging can only be implemented efficiently asynchronous, using solid state drives is irrelevant with regards latency and don't provide any edge. The general advice still applies: eliminate memory allocations, data copies, lock contention and context switching. Write to a ring buffer in shared memory between the thread that logs and the thread that dispatches the message to the kernel, and if you are really skillful you could possible implement it lockless/lockfree.
I'm not sure that I agree that Flash is not irrelevant. If you use a ring buffer as you suggest that the async flush rate matters otherwise you have a serious impedance mismatch on production vs ability to write. ring buffers are great but problematic in scenarios (like logging) where the production rate is often much higher than the drain rate. The trick is to use a stripped drain coupled with DirectIO and a FusionIO style SSD. The stripped drain is so that you can increase the parallelism into the SSD. The boys at RethinkDB have proven this to be very effective for DB's as well.
ReplyDelete