.net - Is there a way to speed up reading the data? -

-

in program created following logic reading data database , storing list<>:

                npgsqlcommand cmd = new npgsqlcommand(query, conn);                 list<userinfo> result = new list<userinfo>();                 npgsql.npgsqldatareader rdr = cmd.executereader();                 while (rdr.read())                 {                     string userid = rdr[0].tostring();                     string sex = rdr[1].tostring();                     string strdatebirth = rdr[2].tostring();                     string zip = rdr[3].tostring();                      userinfo userinfo = new userinfo();                     userinfo.msisdn = userid;                     userinfo.gender = sex;                     try                     {                         userinfo.birthdate = convert.todatetime(strdatebirth);                     }                     catch (exception ex)                     {                     }                     userinfo.zipcode = zip;                     userinfo.demographicsknown = true;                     userinfo.agegroup = getagegroup(strdatebirth);                     if (result.count(x => x.id== userid) == 0)                         result.add(userinfo);                 }

the performance of code poor. there on 2m of records , after half hour list userinfo contains 300.000 records.

does know how speed data reading database?

you using .count when mean .any()
whenever call .count enumerating entire collection see if have single match....

consider question you're asking:
"how many rows have match condition? number equal zero?"

what mean is:
"do rows match condition?"

in context, create hashset of userid values. checking existence in hashset (or dictionary) can faster checking same in list.

furthermore, if do have userid, parsed , read values no reason. check myhashset.contains(userid) first, add.

this primary reason it's slow. n rows you're performing nth-triangular enumerations of collection!

edit: consider untested change: don't know if reader supports typed read methods getstring() if doesn't use had before.

npgsqlcommand cmd = new npgsqlcommand(query, conn); list<userinfo> result = new list<userinfo>(); npgsql.npgsqldatareader rdr = cmd.executereader(); hashset<string> userhash = new hashset<string>(); // int?  while (rdr.read()) {     string userid = rdr.getstring(0);     if (!userhash.contains(userid))     {         string strdatebirth = rdrgetstring(2);         userinfo userinfo = new userinfo();         userinfo.msisdn = userid;         userinfo.gender = rdr.getstring(1);         datetime parseddate; // not used if parse fails         if (datetime.tryparse(strdatebirth, out parseddate))         {             userinfo.birthdate = parseddate;             // userinfo.agegroup = getagegroup(strdatebirth); // why take string?             // rewrite getagegroup method take datetime             userinfo.agegroup = getagegroup(parseddate);         }         userinfo.zipcode = rdr.getstring(3);         userinfo.demographicsknown = true;         result.add(userinfo);         userhash.add(userid);     } }

this keep first instance of user row find (which current code does). if want keep last instance can use dictionary , eliminate .contains() call altogether.

edit: noticed sample never added userid hash... whoops... added in there.


Comments

Post a Comment

Popular posts from this blog

django - How can I change user group without delete record -

-
on website user have 1 group. , user can change group. it's made user.groups.clear() and user.groups.add(new_group) but it's not efficient, because there 2 sql query: delete, insert. how can change group update query? user , group related each other using manytomanyfield . means intersection table exists relating both entities, , if don't specify model map (using through attribute) django creates 1 you. looking @ sources django.contrib.auth.models see that's case. fortunatly, can access intermediary model using through attribute of manager (in case, user.groups.through ). can use regular model. example: >>> alice = user.objects.create_user('alice', 'alice@example.com', 'alicepw') >>> employee = group.objects.create(name='employee') >>> manager = group.objects.create(name='manager') >>> alice.groups.add(employee) >>> alice.groups.all() [<group: employee...
Read more

java - Need to add SOAP security token -

-
i need set custom soap header attribute on jax-ws generated webservice client. case webservice calls must go through proxy server requiring specific token (recieved web request header) present in soap request header. e.g.: 1 carserviceservice service = null; 2 service = new carserviceservice(new url(url), new qname(qname); 3 carserviceendpoint port = service.getcarserviceport(); it seems in line 3 wsdl retrieved , call fails due missing security token. 1 point direction on how done? a detailed example has been mentioned here: creating , deploying jax-ws web service on tomcat 6 this article shows how create , use security token .
Read more

java - EclipseLink JPA Object is not a known entity type -

-
i have strange problem eclipselink , object want persist. have 1 object (keypointlistimpl) stores object keypointimpl in list. persisting keypointimpl objects works great if try persist keypointlistimpl object java.lang.illegalargumentexception says object keypointimpl isn't known entity type. here keypointimpl code: @entity @table(name="keypoints") public class keypointimpl implements keypoint { @id @generatedvalue(strategy = generationtype.identity) private int id; @enumerated(enumtype.string) private detectortype keypointtype; private float x; private float y; private float size; private float angle; private float response; private int octave; private int classid; ... } here keypointlistimpl code: @entity @table(name="keypointlists") public class keypointlistimpl implements keypointlist { @id @generatedvalue(strategy = generationtype.identity) private int id; @onetoone(cascade={ca...
Read more