# Downloader Downloader is the service for seeding/downloading historical data using the BitTorrent protocol. This data is stored in the form of *snapshots*, which are actually immutable `.seg` files. The ETH core instructs the `downloader` component to download (and then seed) specific files from the BitTorrent network. The files are specified by their "info hashes", which are a form of content addressing. The files that ETH core instructs to download are block headers and block bodies. `Downloader` interacts then with the BitTorrent network to retrieve files needed by ETH core. {% hint style="info" %} **Info**: While all Erigon components are separable and can be run on different machines, the `downloader` must run on the same machine as Erigon to be able to share downloaded and seeded files. {% endhint %} ## Start Erigon with snapshots support Like many other Erigon components (txpool, sentry, rpc daemon) `downloader` can be integrated into Erigon or run as a separate process. `Downloader` run by default inside Erigon with the `--snapshots` flag: ```bash ./build/bin/erigon --snapshots --datadir= ``` {% hint style="success" %} **Info**: `--snapshots` flag is compatible with `--prune` flag (more info [here](https://docs.erigon.tech/erigon/v2/basic-usage/usage/type-of-node#full-node)). {% endhint %} ## Running downloader as a separate process It's possible to start `Downloader` as independent process with `--snapshots --downloader.api.addr=127.0.0.1:9093` flag. Before using a separate `downloader` process the executable must be built: ``` cd erigon make downloader ``` And you can then start the `downloader` ```bash ./build/bin/downloader --downloader.api.addr=127.0.0.1:9093 --torrent.port=42068 --datadir= ``` `--downloader.api.addr` - is for internal communication with Erigon `--torrent.port=42068` - is for public BitTorrent protocol listen {% hint style="info" %} You can increase/limit the network usage by adding the following flags: `--torrent.download.rate=512mb --torrent.upload.rate=512mb` `The default download speed is 16mb/sec.` {% endhint %} Erigon on startup sends list of `.torrent` files to `Downloader` and waits for 100% download completion ``` ./build/bin/erigon --snapshots --downloader.api.addr=127.0.0.1:9093 --datadir= ``` Use `--snap.keepblocks=true` to not delete retired blocks from DB. Any network/chain can start with snapshot sync: * node will only download snapshots registered in the next repo * node will move old blocks from DB to snapshots of 1K blocks size, then merge snapshots to bigger range, until snapshots of 500K blocks, then automatically start seeding new snapshot ## Creation of a new network or bootnode You may need to create new snapshots and start seeding them Creating new snapshots will dump blocks from Database to `.seg` files ``` erigon snapshots retire --datadir= ``` Will create the `.torrent` files that `downloader` will automatically seed. The output format is compatible with . {% hint style="info" %} You can change the snapshot size by using the flag `--from=0 --to=1_000_000 --segment.size=500_000` {% endhint %} ``` ./build/bin/downloader torrent_hashes --rebuild --datadir= ``` Start downloader (seeds automatically) ``` ./build/bin/downloader --downloader.api.addr=127.0.0.1:9093 --datadir= ``` {% hint style="info" %} Erigon is not required for snapshots seeding, but Erigon with `--snapshots` also does seeding. {% endhint %} ### Additional info Snapshots creation does not require a fully-synced Erigon, few first stages are enough. For example: STOP\_AFTER\_STAGE=Senders ``` ./build/bin/erigon --snapshots=false --datadir= ``` But for security it is better to have a fully-synced Erigon. Erigon can use snapshots only after indexing them. Erigon will automatically index them but also can run (this step is not required for seeding): ``` ./build/bin/erigon snapshots index --datadir= ``` ## Architecture `Downloader` works based on `/snapshots/*.torrent` files. Such files can be created in 4 ways: * Erigon can do grpc call `downloader.Download(list_of_hashes)`, it will trigger creation of `.torrent` files * Erigon can create new `.seg` file, `Downloader` will scan `.seg` file and create `.torrent` * operator can manually copy `.torrent` files (rsync from other server or restore from backup) * operator can manually copy `.seg` file, Downloader will scan `.seg` file and create .torrent Erigon does: * connect to Downloader * share the list of hashes (see ) * wait for download of all snapshots * when `.seg` file is available it automatically create `.idx` files - secondary indices, for example to find block by hash * then switch to normal staged sync (which doesn't require connection to `Downloader`) * ensure that snapshot downloading happens only once: even if new Erigon version does include new pre-verified snapshot hashes, Erigon will not download them (to avoid unpredictable downtime) - but Erigon may produce them by self. Downloader does: * Read `.torrent` files, download everything described by .torrent files * Use see `./trackers/embed.go` * automatically seeding ### Technical details * To prevent attack `.idx` creation using random Seed - all nodes will have different `.idx` file (and same `.seg` files) * If you add/remove any `.seg` file manually you also need to remove `/snapshots/db` folder ## How to verify that .seg files have the same checksum as current .torrent files Use it if you see strange behavior, bugs, bans, hardware problems, etc. ``` ./build/bin/downloader --verify --datadir= ``` ## Faster rsync ``` rsync -aP --delete -e "ssh -T -o Compression=no -x" ``` ## Release details Start automatic commit of new hashes to branch `master` `crontab -e @hourly cd && ./cmd/downloader/torrent_hashes_update.sh 1>&2 2>> ~/erigon_cron.log` It does push to branch `auto`, before release - merge `auto` to `main` manually ## Command line options To display available options for `downloader` digit: ``` ./build/bin/downloader --help ``` The `--help` flag listing is reproduced below for your convenience. ### Commands {% code overflow="wrap" fullWidth="true" %} ``` snapshot downloader Usage: [flags] [command] Examples: go run ./cmd/downloader --datadir --downloader.api.addr 127.0.0.1:9093 Available Commands: completion Generate the autocompletion script for the specified shell help Help about any command torrent_hashes Flags: --datadir string Data directory for the databases (default "/home/admin/.local/share/erigon") --downloader.api.addr string external downloader api network address, for example: 127.0.0.1:9093 serves remote downloader interface (default "127.0.0.1:9093") --downloader.disable.ipv4 Turns off ipv6 for the downlaoder --downloader.disable.ipv6 Turns off ipv6 for the downlaoder -h, --help help for this command --log.console.json Format console logs with JSON --log.console.verbosity string Set the log level for console logs (default "info") --log.dir.json Format file logs with JSON --log.dir.path string Path to store user and error logs to disk --log.dir.prefix string The file name prefix for logs stored to disk --log.dir.verbosity string Set the log verbosity for logs stored to disk (default "info") --log.json Format console logs with JSON --metrics Enable metrics collection and reporting --metrics.addr string Enable stand-alone metrics HTTP server listening interface (default "127.0.0.1") --metrics.port int Metrics HTTP server listening port (default 6060) --nat string NAT port mapping mechanism (any|none|upnp|pmp|stun|extip:) "" or "none" default - do not nat "extip:77.12.33.4" will assume the local machine is reachable on the given IP "any" uses the first auto-detected mechanism "upnp" uses the Universal Plug and Play protocol "pmp" uses NAT-PMP with an auto-detected gateway address "pmp:192.168.0.1" uses NAT-PMP with the given gateway address "stun" uses STUN to detect an external IP using a default server "stun:" uses STUN to detect an external IP using the given server (host:port) --pprof Enable the pprof HTTP server --pprof.addr string pprof HTTP server listening interface (default "127.0.0.1") --pprof.cpuprofile string Write CPU profile to the given file --pprof.port int pprof HTTP server listening port (default 6060) --torrent.conns.perfile int connections per file (default 10) --torrent.download.rate string bytes per second, example: 32mb (default "16mb") --torrent.download.slots int amount of files to download in parallel. If network has enough seeders 1-3 slot enough, if network has lack of seeders increase to 5-7 (too big value will slow down everything). (default 3) --torrent.maxpeers int unused parameter (reserved for future use) (default 100) --torrent.port int port to listen and serve BitTorrent protocol (default 42069) --torrent.staticpeers string Comma separated enode URLs to connect to --torrent.upload.rate string bytes per second, example: 32mb (default "4mb") --torrent.verbosity int 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (must set --verbosity to equal or higher level and has defeault: 3) (default 2) --trace string Write execution trace to the given file --verbosity string Set the log level for console logs (default "info") --verify Force verify data files if have .torrent files Use " [command] --help" for more information about a command. ``` {% endcode %}