Heavy disc load

kermithermos · #1 (**permalink**) June 7th, 2023

Thank you for your helpful answer:

Maybe I am wrong here and overlooked something:

When I looked at the logs, every 5 sec a write is going on. most of them are 0 bytes written messages. So I searched for 5000 (ms) in the code (DownloadDataWriter) and there you go:
Line 111:

Code:

wait( 5000 );

This should be at least a random value in a range, to spread the time when the threads are served again. They are all set to 5000 and are notified the same time again and most of them wait again 5000ms,...

Every thread waits the same time, that means they wake up at the same time, and later additional comes on top the forced write in Line 137/138. That means: A much higher buffersize with this 5000ms is totally ignored if you have not much downloadtraffic thats why it is never filled to an expected level, its flushed far too soon.
You can use a random value here to "spread" the threads more, set up the forced write time but the inefficient buffer usesage still exists, bec. wait time and downloads speed depend on each other to effeciencely use the buffer. You can use different values for different speeds for improved effiencency but this is still a even more bad hack.
Thats why this time check based implementation IMO is not a good solution.

My idea:
The buffer itself should write data when it reaches the 90% level independent from a time. If buffer reaches 90% enter a data writing queue and pause downloading for this file until buffer is flushed. This is easy to implement and dont have to struggle with different thread situations.

GregorK · #2 (**permalink**) June 8th, 2023

This is how it looks for me now

Code:

230608 10:31:35.824(T60119)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:31:35.824(T60119)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Time for complete write cycle.
230608 10:31:35.824(T60119)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:31:35.824(T60119)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@2c4947fa, amount: 225280 bytes.
230608 10:31:35.837(T60132)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@16e84b3f, amount: 16384 bytes.
230608 10:31:35.840(T60135)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@5294c4, amount: 16384 bytes.
230608 10:31:35.843(T60138)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@7007e514, amount: 893365 bytes.
230608 10:31:35.881(T60176)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@384c68bd, amount: 632250 bytes.
230608 10:31:35.900(T60195)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@21fede92, amount: 331776 bytes.
230608 10:31:35.910(T60205)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@79af60e, amount: 613260 bytes.
230608 10:31:35.927(T60222)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@6233ef0d, amount: 540672 bytes.
230608 10:31:35.940(T60235)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 3269371 bytes.
230608 10:31:35.940(T60235)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total data written was: 3269371 bytes.
230608 10:31:35.941(T60236)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:31:40.946(T65241)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:31:40.946(T65241)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:31:40.946(T65241)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 327158 bytes.
230608 10:31:40.946(T65241)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:31:40.946(T65241)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:31:45.958(T70253)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:31:45.958(T70253)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:31:45.958(T70253)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 641733 bytes.
230608 10:31:45.958(T70253)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:31:45.958(T70253)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:31:50.959(T75254)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:31:50.959(T75254)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:31:50.959(T75254)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 943725 bytes.
230608 10:31:50.959(T75254)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:31:50.959(T75254)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:31:55.965(T80260)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:31:55.965(T80260)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:31:55.965(T80260)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1258300 bytes.
230608 10:31:55.965(T80260)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:31:55.965(T80260)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:00.970(T85265)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:00.970(T85265)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:00.970(T85265)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1396661 bytes.
230608 10:32:00.970(T85265)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:00.970(T85265)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:05.984(T90279)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:05.984(T90279)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:05.985(T90280)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1472691 bytes.
230608 10:32:05.985(T90280)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:05.985(T90280)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:10.997(T95292)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:10.997(T95292)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:10.997(T95292)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1548721 bytes.
230608 10:32:10.997(T95292)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:10.997(T95292)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:16.002(T100297)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:16.002(T100297)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:16.002(T100297)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1568316 bytes.
230608 10:32:16.002(T100297)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:16.002(T100297)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:21.010(T105305)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:21.010(T105305)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:21.010(T105305)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1644092 bytes.
230608 10:32:21.010(T105305)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:21.010(T105305)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:26.024(T110319)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:26.024(T110319)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:26.024(T110319)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1715772 bytes.
230608 10:32:26.024(T110319)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:26.024(T110319)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:31.029(T115324)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:31.029(T115324)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:31.029(T115324)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1818172 bytes.
230608 10:32:31.029(T115324)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:31.029(T115324)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:36.037(T120332)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:36.037(T120332)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Time for complete write cycle.
230608 10:32:36.037(T120332)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:36.037(T120332)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@1134a9e2, amount: 147456 bytes.
230608 10:32:36.041(T120336)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@7007e514, amount: 171595 bytes.
230608 10:32:36.046(T120341)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@384c68bd, amount: 653894 bytes.
230608 10:32:36.076(T120371)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@79af60e, amount: 844916 bytes.
230608 10:32:36.095(T120390)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Trigger buffer write for phex.download.swarming.SWDownloadFile@6233ef0d, amount: 311 bytes.
230608 10:32:36.096(T120391)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 1818172 bytes.
230608 10:32:36.096(T120391)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total data written was: 1818172 bytes.
230608 10:32:36.096(T120391)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:41.106(T125401)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:41.106(T125401)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:41.106(T125401)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 0 bytes.
230608 10:32:41.106(T125401)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:41.106(T125401)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...
230608 10:32:46.111(T130406)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Woke...
230608 10:32:46.111(T130406)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: MaxWriteBufferPerDownload 9437184 - MaxTotalDownloadWriteBuffer 37748736
230608 10:32:46.111(T130406)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Total buffered data was: 299801 bytes.
230608 10:32:46.111(T130406)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: No data was written
230608 10:32:46.111(T130406)(156) DEBUG/phex.download.DownloadDataWriter [DownloadDataWriter]:: Waiting...