TID: Mystic BBS 1.12 A48   
   MSGID: 2:201/137 c6a56385   
   REPLY: 1:3634/12.73 624dd22c   
   TZUTC: 0200   
   Hello Mark!   
      
   Thank you for your reply!   
      
   On 06 Apr 2022, mark lewis said the following...   
    ml> if the cd succeeds they do an ls to see if there's file and directories   
    ml> in it... if there are, they download them and continue walking through   
    ml> the directories...   
      
   OK! Thanks!   
      
    ml>  BW> or if they just conclude "not a file" and skip it. (According to the   
    ml>  BW> spec, they shouldn't look for anything inside that directory, if it   
    ml>  BW> exists.)   
    ml>    
    ml> what spec is that?   
      
   I was thinking of this one:   
      
   https://developers.google.com/search/docs/advanced/robots/robots_txt#examples-o   
   f-valid-robots.txt-urls   
      
   http://example.com/folder/robots.txt   
   "Not a valid robots.txt file. Crawlers don't check for robots.txt files in   
   subdirectories. "   
      
   Best regards   
   Bj”rn   
      
   --- Mystic BBS v1.12 A48 2022/03/26 (Linux/64)   
    * Origin: Star Collision BBS, Uppsala, Sweden (2:201/137)   
   SEEN-BY: 1/123 15/0 90/1 103/705 105/81 106/201 120/340 123/131 124/5016   
   SEEN-BY: 129/305 330 331 153/757 7715 154/10 201/0 137 203/0 124 218/700   
   SEEN-BY: 218/840 221/0 1 6 242 360 226/30 227/114 229/110 206 307   
   SEEN-BY: 229/317 400 424 426 428 452 550 664 700 230/0 240/5832 266/512   
   SEEN-BY: 280/464 5003 282/1038 292/854 8125 301/1 317/3 320/219 322/757   
   SEEN-BY: 335/364 341/66 234 342/200 396/45 423/81 460/58 633/280 712/848   
   SEEN-BY: 2452/250 3634/24 4500/1   
   PATH: 201/137 0 203/0 280/464 221/1 6 229/664 426