--------------------------------------------------------------------------------------------------------------------- TESTS LOCAL --------------------------------------------------------------------------------------------------------------------- http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://bot.sannysoft.com/ http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://www.scrapingcourse.com/cloudflare-challenge http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://arh.antoinevastel.com/bots/areyouheadless http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://www.browserscan.net/bot-detection http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://fingerprint.com/products/bot-detection/ http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://fr.indeed.com/cmp/Institut-Vendôme-2 --------------------------------------------------------------------------------------------------------------------- TESTS REMOTE --------------------------------------------------------------------------------------------------------------------- https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://bot.sannysoft.com/ https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://www.scrapingcourse.com/cloudflare-challenge https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://arh.antoinevastel.com/bots/areyouheadless https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://www.browserscan.net/bot-detection https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://fingerprint.com/products/bot-detection/ https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://fr.indeed.com/cmp/Institut-Vendôme-2 --------------------------------------------------------------------------------------------------------------------- LOCALHOST BASIC URLS --------------------------------------------------------------------------------------------------------------------- TEST RANDOM URLS WITH JS AND PROXY http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://www.pharow.com http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.dropcontact.com/fr http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://payfit.com/fr/ http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.lequipe.fr/explore-video/020-la-barkley-sans-pitie http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.afjv.com/societe/547-virtuos.htm http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://fr.indeed.com/cmp/Umanis --------------------------------------------------------------------------------------------------------------------- REMOTE BASIC URLS --------------------------------------------------------------------------------------------------------------------- TEST RANDOM URLS WITH JS AND PROXY https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://www.pharow.com https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://www.dropcontact.com/fr https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://payfit.com/fr/ https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://www.lequipe.fr/explore-video/020-la-barkley-sans-pitie https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://www.afjv.com/societe/547-virtuos.htm https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=false&url=https://fr.indeed.com/cmp/Umanis --------------------------------------------------------------------------------------------------------------------- RAW URLS WITH ISSUES --------------------------------------------------------------------------------------------------------------------- NOT WORKING URLs works on localhost, not on remote server : http://www.francoispneus.com Timeout - never resolved https://poledentaire-lacourneuve.fr/ Blocked 406 Mime error http://www.cab15.fr Execution context was destroyed (ok avec smartproxy) https://www.ets-marin.fr/ Will return a 404 on cthulhu / but a 200 in localhost or using proxy https://www.lacouetterie.fr 564294 http://www.montaigne-immo-21.com/legal.html --------------------------------------------------------------------------------------------------------------------- LOCALHOST URLS WITH ISSUES --------------------------------------------------------------------------------------------------------------------- Will return a 404 on cthulhu / but a 200 in localhost even using proxy http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://www.optiplus.io Will return a cloudflare 503 on cthulhu possible solution here : https://stackoverflow.com/questions/68930114/bypass-cloudflares-captcha-with-headless-chrome-using-puppeteer-on-heroku https://www.ballouchi.com 3411 Works with load event but not networkidle2 http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://pranaloe.fr TEST LINKEDIN - SANS PROXY http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=true&url=https://fr.linkedin.com/in/julie-lafleur-47899a179 TEST BOT http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://bot.sannysoft.com/ http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://bot.incolumitas.com/ TEST PASS CLOUDFLARE http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://nowsecure.nl TEST USER AGENTS http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.whatismybrowser.com/detect/what-is-my-user-agent/ OTHER TEST http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://bot.incolumitas.com/ http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.fnac.com TEST PROXY / IP ADRESS http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.whatismyip.com/fr/ http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://whatismyipaddress.com/fr/mon-ip CDISCOUNT http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.cdiscount.com/mpv-2-game-s.html TEST JS (attention : il faut désactiver js dans le browser aussi) http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=false&show_page=true&url=https://www.whatismybrowser.com/fr/detect/is-javascript-enabled TEST 404 http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=false&url=http%3A%2F%2Fwww.xeline-production.fr%2Fcontacts.php TEST REDIR http://localhost:7472/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=false&use_js=true&show_page=false&url=https%3A%2F%2Fwww.airfrance.fr TEST BLOCKED RESSOURCE http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https%3A%2F%2Fwww.cdiscount.com%2Fmpv-195-jardideco.html http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https%3A%2F%2Ffr.indeed.com%2Femplois%3Fq%3DCaisses%26l%3Dloiret%26radius%3D0%26limit%3D50 http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https%3A%2F%2Ffr.indeed.com%2Fviewjob%3Fjk%3D1927435a26228ddc ERROR WITH VIDEO (video keep being blocked and retried) http://localhost:7472?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&url=https://store.steampowered.com/app/1548810/illegal_Transporter_Simulator/ */ CRAWL https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://bot.sannysoft.com/ test cloudflare ! https://crawl.pharow.com/?api_key=1c65ff5837efd1d55e8!**eee1a0125fc05&use_proxy=true&use_js=true&show_page=true&url=https://www.scrapingcourse.com/cloudflare-challenge