Replace pwrite with asynchronous write

ASD uses synchronous reads/writes for IOs on disk/file devices. During startup ASD creates a thread per storage backend. The thread consumes a single ssd_write_buf from queue and issues a synchronous write to disk. As a result, only a single write IO can be outstanding at a time.

The size of the Q from which writer thread consumes writes is limited. If the writer thread is not able to flush at the rate of client, ASD start issuing OVERLOAD errors.

This is a first attempt to issue asynchronous writes to device, raising the pull request just to get feedback. The work is not complete, however I would like to get early review.

I have written a benchmark tool, which write 4096 sized objects with various IO depths (from 1 to 128). It always fails with OVERLOAD error. Besides, while the benchmark is running, IOSTAT reports only single outstanding IO to disk.

The overload error is gone with the modified code and the IO Request Q depth reported by IOSTAT is continuously more than 1.

The code change includes

1. During startup for each SSD device
   • create a additional cf_poll, register a eventfd for AsyncIO
   • create a separate eventfd for flush_write_q an shadow_write_q. Both the eventfds are registered with same poll.
2. Removed the thread which serves shadow write q. Create a single thread per SSD for all writes
3. signal eventfd, when a ssd_write_buf is pushed into any q.
4. The writer thread
   • reads as many ssd_write_buf as possible, fills IOCB, and issues multiple writes in a single system call.
   • The writer thread also handle the write completions

Signed-off-by: Prasad Joshi prasadjoshi.linux@gmail.com