TID: FMail-lnx64 2.1.0.18-B20170815   
   RFC-X-No-Archive: Yes   
   TZUTC: 0200   
   CHRS: UTF-8 2   
   PID: GED+LNX 1.1.5-b20161221   
   MSGID: 2:280/464 6065cae8   
   REPLY: 16159.fido-synchron@1:3634/12 24cb32b5   
   * Originally in SYNCHRONET   
   * Crossposted in NET_DEV   
      
   Hi mark,   
      
   On 2021-04-01 08:33:26, you wrote to me:   
      
    ml> in any case, dupe checking in FTN is not done my /just/ detecting   
    ml> duplicate MSGIDs and rejecting the others... the header, including the   
    ml> time stamp, as well as the message body should be taken into   
    ml> account...   
      
   That's how FMail does it.   
      
    ml> it should also be said that CRC16/CRC32 on the message bodies is also   
    ml> not sifficient... even with filtering out white space and the various   
    ml> EoLs... this because, and most programmers know this, there's a   
    ml> limited supply of CRC values in the tables and it is all too easy to   
    ml> find "hash clashes"... CRC16 has only 65536 values... CRC32 has only   
    ml> 4294967296 values... the "Birthday Problem" also comes into play...   
      
   Indeed. That's why it's on my todo list to go from 32 bit to 64 bit hash   
   values in FMail. That would be more than enough to keep a (very) big dupe   
   "database", and still have a very small probability for hash collisions.   
      
    ml> these days, MD5 and SHA1 are also out due to defects in them...   
      
   That's because of their security aspect. That's not an issue when you would be   
   using them as hash's for dupe detection.   
      
    ml> SHA256 would be the first really useful algorithm or SHA512...   
      
   Using a 256 or even 512 bit secure hash value would be overkill for dupe   
   detection. And would be using way too much resources to calculate and check   
   them.   
      
   64 Bit is enough, and it doesn't have to be secure. My prime candidate for   
   FMail is this one:   
      
   https://github.com/Cyan4973/xxHash   
      
    ml> but the real key is to filter out the stuff that can change and hash   
    ml> only that which won't...   
      
   Indeed.   
      
    ml> anyway, i'm done with this topic in this area... the discussion really   
    ml> belongs elsewhere for those that are truely interested in implementing   
    ml> proper duplicate detection in FTNs...   
      
   (I've crossposted to NET_DEV.)   
      
      
   Bye, Wilfred.   
      
   --- FMail-lnx64 2.1.0.18-B20170815   
    * Origin: FMail development HQ (2:280/464)   
   SEEN-BY: 1/123 18/200 90/1 103/705 105/81 120/340 123/131 124/5016   
   SEEN-BY: 154/10 203/0 221/0 226/30 227/114 229/424 426 664 700 1016   
   SEEN-BY: 229/1017 240/5138 5411 5824 5832 5853 249/206 317 280/464   
   SEEN-BY: 280/5003 282/1038 288/100 292/854 8125 310/31 317/3 322/757   
   SEEN-BY: 342/200 396/45 423/120 633/280 712/848 770/1 2432/390 2454/119   
   PATH: 280/464 240/5832 229/426