Can boost.asio read/write io of files asynchronously?
As title, I read boost.asio documents and generally said io about http/network. I don't see any info. of io of files. Does boost.asio support read/write io of files asynchronously?
On 31/08/2020 14:59, lampahome wrote:
As title, I read boost.asio documents and generally said io about http/network.
I don't see any info. of io of files. Does boost.asio support read/write io of files asynchronously?
I haven't tried any of these methods, but it does provide the posix::stream_descriptor and windows::{stream,random_access}_handle classes which should be able to operate on native files. (You'll have to use a native OS-level function to open the handle/descriptor for the file and pass ownership to the ASIO object; they lack means to do the opening itself.) https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/overview/posix/str... https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/overview/windows/s... https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/overview/windows/r... Alternatively, there's another library (that I also haven't used) dedicated to async file I/O. It originally started life based on ASIO but I think it's since diverged into something else. And called something different from the last time I saw it as well. Its current home appears to be: https://ned14.github.io/llfio/ (The impression I have, which may be mistaken, is that it is possibly overkill for general application use, unless you're trying to implement some kind of custom database engine rather than just using an off-the-shelf DB library.)
I haven't tried any of these methods, but it does provide the posix::stream_descriptor and windows::{stream,random_access}_handle classes which should be able to operate on native files. (You'll have to use a native OS-level function to open the handle/descriptor for the file and pass ownership to the ASIO object; they lack means to do the opening itself.)
Thank u. That gives me a direction. The only sad thing is very little document/examples described about this API of boost.asio .
On Mon, 31 Aug 2020 at 15:01, lampahome via Boost-users < boost-users@lists.boost.org> wrote:
I haven't tried any of these methods, but it does provide the
posix::stream_descriptor and windows::{stream,random_access}_handle classes which should be able to operate on native files. (You'll have to use a native OS-level function to open the handle/descriptor for the file and pass ownership to the ASIO object; they lack means to do the opening itself.)
Thank u. That gives me a direction. The only sad thing is very little document/examples described about this API of boost.asio .
This is a common complaint of Asio. If you would like to tap into a great body of knowledge and interest in asio/networking-ts you can do so by joining the #beast channel in cpplang slack https://cppalliance.org/slack/ Beast depends very heavily on Asio and we have worked to build a strong body of expertise.
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Richard Hodges hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212
If you would like to tap into a great body of knowledge and interest in asio/networking-ts you can do so by joining the #beast channel in cpplang slack
If I'm interest in asynchronous io of file, do I try asio? What I look for is like libaio in unix system, but still found no wrapper/library to do like that. thanks
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O. On Tue, 1 Sep 2020 at 06:40, lampahome via Boost-users < boost-users@lists.boost.org> wrote:
If you would like to tap into a great body of knowledge and interest in
asio/networking-ts you can do so by joining the #beast channel in cpplang slack
If I'm interest in asynchronous io of file, do I try asio? What I look for is like libaio in unix system, but still found no wrapper/library to do like that.
thanks _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Richard Hodges hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212
On 01/09/2020 11:54, Richard Hodges via Boost-users wrote:
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O.
Unless you were using a custom io_uring based i/o service, all your asynchronous file i/o on Linux was actually synchronous all along. Linux doesn't implement async file i/o until io_uring, and even then, not properly until kernel 5.5 and even then again, not really until kernel 5.9 releases. Some would argue that proper, reliable, sans-pathological-corner-case async file i/o won't arrive in Linux until some time in the 6.x kernel series. Niall
Hello, Niall Douglas Wrote:
Unless you were using a custom io_uring based i/o service, all your asynchronous file i/o on Linux was actually synchronous all along.
It doesn't really matter, whether the IO really is performed asynchronously. The asio API is asynchronously in the sense, that the completion handler is not called from within the async_*() call but later from within the main loop and the main loop does not block, while waiting for the IO operation to complete. It just happens to be the same Linux API, that is used for async socket IO and for async file io, since on linux systems, sockets are files. But: Things probably are different on windows systems, so there is no guarantee. 73, Mario -- Mario Klebsch Actia I+ME GmbH Mario.klebsch@ime-actia.de Dresdenstrasse 17/18 Fon: +49 531 38 701 716 38124 Braunschweig Fax: +49 531 38 701 88 German
On 01/09/2020 15:30, Klebsch, Mario via Boost-users wrote:
Niall Douglas Wrote:
Unless you were using a custom io_uring based i/o service, all your asynchronous file i/o on Linux was actually synchronous all along.
It doesn't really matter, whether the IO really is performed asynchronously. The asio API is asynchronously in the sense, that the completion handler is not called from within the async_*() call but later from within the main loop and the main loop does not block, while waiting for the IO operation to complete.
It just happens to be the same Linux API, that is used for async socket IO and for async file io, since on linux systems, sockets are files.
No, on Linux you cannot use epoll() with file descriptors which refer to seekable devices. You get an error. As the ASIO i/o service on Linux uses epoll(), one quite literally cannot multiplex i/o on files on Linux with ASIO without a custom i/o service implementation. If file i/o appears to work on ASIO on Linux, then ASIO is never handing those file descriptors to epoll() at all, and is always blocking on reads and writes. In this situation, it would be more efficient to call the preadv() and pwritev() syscalls directly. Niall
Richard Hodges via Boost-users <boost-users@lists.boost.org> 於 2020年9月1日 週二 下午6:54寫道：
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O.
Is there any function can handle the finished IO operation?
Assume I have two threads. One is only submit async io to file, and another one is only handle event of those executed io. I only find handler function but wonder if any easier handler to handle all. thanks
On Wed, 2 Sep 2020 at 06:59, lampahome <pahome.chen@mirlab.org> wrote:
Richard Hodges via Boost-users <boost-users@lists.boost.org> 於 2020年9月1日 週二 下午6:54寫道：
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O.
Is there any function can handle the finished IO operation?
Assume I have two threads. One is only submit async io to file, and another one is only handle event of those executed io. I only find handler function but wonder if any easier handler to handle all.
I'm not quite sure why you'd need two threads. But if you did, asio allows you to bind handlers to executors prior to submission to asynchronous initiating functions. The handler will be executed by its associated executor, which could if you require be running in a different thread. e.g. async_write(f, buf, asio::bind_executor(other_executor, [](error_code ec, std::size_t written) { // handler now executed by other_executor }); If submitting across threads you'd want to be sure to protect access to the file object with a mutex or a strand.
thanks
-- Richard Hodges hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212
On 31/08/2020 05:08, Gavin Lambert via Boost-users wrote:
Alternatively, there's another library (that I also haven't used) dedicated to async file I/O. It originally started life based on ASIO but I think it's since diverged into something else. And called something different from the last time I saw it as well.
Its current home appears to be: https://ned14.github.io/llfio/
Yes, one part of it should enter the C++ 23 standard, other parts may do so. LLFIO doesn't export publicly any async file i/o implementation however. This is because async file i/o usually has worse performance than synchronous file i/o. So you generally want to use synchronous i/o tightly integrated with kernel threads, if concurrency is desired (quite often it is not).
(The impression I have, which may be mistaken, is that it is possibly overkill for general application use, unless you're trying to implement some kind of custom database engine rather than just using an off-the-shelf DB library.)
Spot on. The only place where async i/o makes a lot of sense is where i/o latencies are unpredictable most of the time. This is usually the case with socket i/o latencies, it is rarely the case with file i/o (and as you mention, only in very high random access use cases across datasets far exceeding RAM does disabling caching for file i/o make sense). Niall
participants (5)
-
Gavin Lambert -
Klebsch, Mario -
lampahome -
Niall Douglas -
Richard Hodges