On 2017 Jan 08 10:30:12, you wrote to me:   
      
    NA> On 08 Jan 17  09:16:24, Mark Lewis said the following to Nick Andre:   
      
    ML>> exactly sure on the details but i know that they greatly influenced me   
    ML>> to my software place the MSGID as close to the beginning of the control   
    ML>> lines possible so that db would not detect messages posted within one   
    ML>> second as dupes... this was especially important when testing at 100+   
    ML>> posts per second... are you willing to share information about how db   
    ML>> does its dupe detection so others can understand more? please?   
      
    NA> Its not that hard to understand. A CRC is computed from the header and   
   date   
    NA> of the tossed message. I would have to dig into the code and I'm not sure   
    NA> how many bytes are being included from the start of the message.   
      
   yeah, these little details are what is/was being sought... if the header and X   
   bytes are being read into a buffer and then the CRC calculated on that entire   
   raw buffer or if each field is read individually and then fed to the CRC   
   calculator... knowing this may help others who are trying to work out how to   
   do dupe checking that doesn't rely on MSGID alone... that because messages   
   without MSGID can'tbe checked that way so an alternative or three is   
   desired/needed...   
      
    NA> Each Echomail area has a cache database file. In the case of *.MSG,   
    NA> this is called DBRIDGE.DUP and resides in each area and for   
    NA> Hudson/QBBS there is one database segmented slightly different. The   
    NA> CRC's are kept in there. I believe the code sets the cache database   
    NA> size at 1,024 entries.   
      
   i remember the different dup cache files... i didn't know they were limited to   
   a paltry 1024 entries, though... i never dug that deep ;)   
      
    NA> Interestingly it appears that there is a "reputation" method for the   
    NA> cache database. It appears as it is loaded into RAM during a toss, any   
    NA> time a CRC match is encountered, that CRC is pushed up the cache   
    NA> table, while CRC's of legitimate messages end up being pushed down.   
    NA> The CRC table is saved into that cache file every time the Echomail   
    NA> area changes in the toss cycle; or there are no more packets to toss.   
      
   that's pretty interesting... i guess that's so that messages with more dupes   
   can be detected faster with their CRCs at the top of the queue... interesting   
   idea and i'm sure one that was important back in the day of slower machines :)   
      
   )\/(ark   
      
   Always Mount a Scratch Monkey   
   Do you manage your own servers? If you are not running an IDS/IPS yer doin' it   
   wrong...   
   ... Actually, if they leak, you've pumped them too many times.   
   ---   
    * Origin:  (1:3634/12.73)