TDvorak/FacrScraper
1
0
Fork 0
mirror of https://github.com/Dvorinka/facr-scraper.git synced 2026-06-04 04:22:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(scraper): implement CloakBrowser support and enhance request stealth

Browse Source 
Integrate CloakBrowser to improve success rates against Cloudflare
challenges and implement more robust request handling in the Go backend.

- Add CloakBrowser integration to Dockerfile and requirements
- Implement domain-specific request semaphores in Go to prevent rate-limiting
- Add shared HTTP client with cookie jar and header preservation for
  better session management
- Enhance request headers in Go to include modern client hints (Sec-Ch-Ua)
- Add benchmarking scripts to compare fetch methods (urllib vs Scrapling
  vs CloakBrowser)
- Update docker-compose to support CloakBrowser environment variables
- Optimize Docker image by pre-downloading patched Chromium binaries
This commit is contained in:
Tomas Dvorak
2026-05-17 17:52:52 +02:00
parent aa47f4309f
commit ed61d8ab8e
12 changed files with 608 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockerfile.patch
+6
View File
@@ -0,0 +1,6 @@
FROM facr-scraper:cloakbrowser
USER root
COPY facr-scraper-new /usr/local/bin/facr-scraper
COPY scripts/cloakbrowser_fetch.py /opt/scrapling/scripts/cloakbrowser_fetch.py
RUN chmod +x /usr/local/bin/facr-scraper && chown -R scraper:scraper /opt/scrapling/scripts
USER scraper