lhbelfanti/goxcrap repository overview

⁠GoXCrap

X (formerly Twitter) web scrapper, written in Go.

⁠How to use it

Add the following environment variables to the docker-compose.yml, and replace the values within parenthesis

⁠App settings

APP_EXPOSED_PORT=(GoXCrap Host Port)
APP_INTERNAL_PORT=(GoXCrap Container Port)

⁠Scrapper settings

EMAIL=(Twitter account email)
USERNAME=(Twitter username)
PASSWORD=(Twitter password)
BROKER_CONCURRENT_MESSAGES=(Number of concurrent messages that will be processed)

⁠Selenium Chrome driver paths

DRIVER_PATH=(The path to the Chrome driver) --> Example: /usr/bin/chromedriver
BROWSER_PATH=(The path to the Chrome browser) --> Example: /usr/bin/chromium

⁠RabbitMQ settings

RABBITMQ_USER=(The RabbitMQ user)
RABBITMQ_PASS=(The RabbitMQ password)
RABBITMQ_PORT=(The RabbitMQ port) --> Usually 5672

⁠External APIs URLs

SAVE_TWEETS_API_URL=(Domain of the application with the endpoint /tweets/v1)

⁠More info

https://github.com/lhbelfanti/goxcrap⁠