linux - How to prioritize write() over mmap updates (or delay mmap page cache flush) -

-

i'm running specialized db daemon on debian-64 64g of ram , lots of disk space. uses on-disk hashtable (mmaped) , writes actual data file regular write() calls. when doing lot of updates, big part of mmap gets dirty , page cache tries flush disk, producing lots of random writes in turn slows down performance of regular (sequential) writes data file.

if possible delay page cache flush of mmaped area performance improve (i assume), since several (or all) changes dirty page written @ once instead of once every update (worst case, in reality of course aggregates lot of changes anyway).

so question: possible delay page cache flush memory-mapped area? or possible prioritze regular write? or have other ideas? madvise , posix_fadvise don't seem make difference...

you play tuneables in /proc/sys/vm. example, increase value in dirty_writeback_centisecs make pdflush wake less often, increase dirty_expire_centiseconds data allowed stay dirty longer until must written out, , increase dirty_background_ratio allow more dirty pages stay in ram before must done.
see here comprehensive description of values do.

note affect every process on machine, seeing how you're running huge database server, chances no problem since don't want else run on same machine anyway.

now of course delays writes, still doesn't solve problem of dirty page writebacks competing write (though collapse few writes if there many updates).
but: can use sync_file_range syscall force beginning write-out of pages in given range on "write" file descriptor (sync_file_range_write). while dirty pages written @ unknown time later (and greater grace periods), manually kick off writeback on ones you're interested.
doesn't give guarantees, should work.

be sure absolutely positively read documentation, better read twice. sync_file_range can corrupt or lose data if use wrong. in particular, must sure metadata up-to-date , flushed if appended file, or data has been "successfully written" "gone" in case of crash.


Comments

Post a Comment

Popular posts from this blog

django - How can I change user group without delete record -

-
on website user have 1 group. , user can change group. it's made user.groups.clear() and user.groups.add(new_group) but it's not efficient, because there 2 sql query: delete, insert. how can change group update query? user , group related each other using manytomanyfield . means intersection table exists relating both entities, , if don't specify model map (using through attribute) django creates 1 you. looking @ sources django.contrib.auth.models see that's case. fortunatly, can access intermediary model using through attribute of manager (in case, user.groups.through ). can use regular model. example: >>> alice = user.objects.create_user('alice', 'alice@example.com', 'alicepw') >>> employee = group.objects.create(name='employee') >>> manager = group.objects.create(name='manager') >>> alice.groups.add(employee) >>> alice.groups.all() [<group: employee...
Read more

java - Need to add SOAP security token -

-
i need set custom soap header attribute on jax-ws generated webservice client. case webservice calls must go through proxy server requiring specific token (recieved web request header) present in soap request header. e.g.: 1 carserviceservice service = null; 2 service = new carserviceservice(new url(url), new qname(qname); 3 carserviceendpoint port = service.getcarserviceport(); it seems in line 3 wsdl retrieved , call fails due missing security token. 1 point direction on how done? a detailed example has been mentioned here: creating , deploying jax-ws web service on tomcat 6 this article shows how create , use security token .
Read more

java - EclipseLink JPA Object is not a known entity type -

-
i have strange problem eclipselink , object want persist. have 1 object (keypointlistimpl) stores object keypointimpl in list. persisting keypointimpl objects works great if try persist keypointlistimpl object java.lang.illegalargumentexception says object keypointimpl isn't known entity type. here keypointimpl code: @entity @table(name="keypoints") public class keypointimpl implements keypoint { @id @generatedvalue(strategy = generationtype.identity) private int id; @enumerated(enumtype.string) private detectortype keypointtype; private float x; private float y; private float size; private float angle; private float response; private int octave; private int classid; ... } here keypointlistimpl code: @entity @table(name="keypointlists") public class keypointlistimpl implements keypointlist { @id @generatedvalue(strategy = generationtype.identity) private int id; @onetoone(cascade={ca...
Read more