REPLY: 1:104/57.0 4ec959e8   
   MSGID: 1:3634/12.73 5d7e5a34   
   PID: GED+LNX 1.1.5-b20180707   
   CHRS: CP437 2   
   TZUTC: -0400   
   TID: hpt/lnx 1.9.0-cur 17-02-17   
      
    On 2019 Sep 15 08:31:20, you wrote to me:   
      
    JM> -> MSGID is the main way but older software doesn't generate MSGID so   
    JM> -> other methods need to be used...   
      
    JM> My mailer/tosser uses a combined approach.  If the message contains a   
    JM> MSGID then use its value, otherwise CRC the header and message body   
    JM> including control lines but never the SEEN-BY/PATH lines (considering   
    JM> they change all the time).   
      
   this is good but for one small thing... there is a package that is known to be   
   reformatting messages in transit which is going to throw the message body CRC   
   out the door... there is no estimate on when this but will be fixed as the   
   developer is apparently quite busy with RL outside of FTNs...   
      
    JM> The tosser never duplicates an MSGID either as it maintains a file   
    JM> with the last used value seeded upon creation by the current   
    JM> date/time.  This prevents issues should that file get deleted.   
      
   sounds similar to what my MSGID code does... i've shared that information with   
   several folks... not sure if you were one of those or not... i still have the   
   original 1994 (i think) post that described it, too :)   
      
    JM> To provide speed and limit disk space, I also have an expiration   
    JM> mechanism (user configurable) that will purge CRC entries after a given   
    JM> amount of time (ie: 2 weeks but not more than 30 days).  So while it's   
    JM> efficient catching dupes in that time period, if someone does a rescan   
    JM> and dumps everything back into the echo a month later, it won't catch   
    JM> them. It's a trade off, but back in the day when we had 40mb drives and   
    JM> 8088/80286 processors, it was extremely important.   
      
   yeah and that's gonna likely be a problem since the spec states three years...   
   in this day in time, retaining three years worth of dupe detection data should   
   be a small drop in the bucket of available drive space and processing power   
   needed to perform a lookup...   
      
    JM> -> instead of CRC... the problem then comes from those systems that   
    JM> -> mistakenly reformat the messages as they process them and write the   
    JM> -> reformatted messages to new PKTs... now the message body is   
      
    JM> Yeah this is and always will be an issue.   
      
   not if the message body is not CRC'd ;)   
      
   i really like (IIRC) the d'bridge method of taking the header and first 40   
   bytes (i think) of the message body to get those few initial control lines and   
   using that... that'll take care of the different dates as well as the MSGID   
   but i would also grab the MSGID if it exists and store it in the database as   
   well... basically i'm thinking of at least two or three fields in each   
   record...   
      
    JM> -> is apparent on systems that only get, for example, one posting of   
    JM> an   
    JM> -> echos rules each month and only accept new postings of those rules   
      
    JM> It would seem to me, (me mind you) that if you're moderating an echo,   
    JM> your software "should" be able to generate a MSGID to prevent this issue   
    JM> entirely.  But hey...   
      
   that depends on the software used... some text file posting tools are really   
   old and do not have any concept of MSGID... i'm thinking of the old Harvey's   
   Robot in at least one case...   
      
    JM> -> what i would do would be to ask other tosser devs what they use in   
    JM> -> their code...   
    JM> ->   
    JM> -> listed in no particular order:   
    JM> ->   
    JM> -> tobias burchhardt  - fastecho   
    JM> -> rob swindell       - sbbsecho   
    JM> -> nick andre         - d'bridge   
    JM> -> vince coen         - mbse's tosser   
    JM> -> kim heino          - bbbs' tosser   
    JM> -> wilfred van velzen - fmail   
    JM> -> james coyle        - mystic   
      
    JM> Thanks Mark...   
      
   you're welcome... i hope that you've also seen the other two posts about HPT   
   and intermail which should also be added to the above list...   
      
   )\/(ark   
      
   Once men turned their thinking over to machines in the hope that this would   
   set them free. But that only permitted other men with machines to enslave them.   
   ... You may never know who's right but you always know who is in charge!   
   ---   
    * Origin:  (1:3634/12.73)   
   SEEN-BY: 1/120 123 14/6 15/2 18/0 123/0 25 120 150 755 135/300 153/757   
   SEEN-BY: 153/7715 227/114 229/354 426 1014 240/5832 249/206 317 261/38   
   SEEN-BY: 280/464 300/4 317/3 322/757 342/200 633/0 267 280 281 412   
   SEEN-BY: 633/509 640/1321 1384 712/620 848 770/1 3634/0 12 15 24 119   
   PATH: 3634/12 640/1384 712/848 633/280 229/426