Syncing the full blockchain from scratch with Geth is no longer possible

So I've been trying to sync to the full blockchain from scratch with Geth for over a week, and towards the end it's too slow to ever catch up to the current block. I don't think the developers have tried syncing from scratch in a while, because AFAIK it's no longer possible, at least on average consumer grade hardware. I've tried --fast and increasing the cache to 4096, but even that slows to a crawl toward the end and the blocks are processed, at best, at the same speed as the new ones coming in. It's usually a bit slower.

Anyone want to help out on this one? I realize one can sync quickly with the --light blockchain, but some of us want the real mccoy.

Comments

  • toddtodd Member Posts: 2
    It still works as of Dec 2017.
    It requires SSD with space available - presently 102Gb. and it takes a few days, anywhere from 2 to 10 days. My internet speed is 150mbps and wired.
    Just regular client without any flags.
    Now I run it every 3-4 days and it takes around 3-4 hours to synchronise again.
  • chromachroma Member Posts: 8
    edited January 2018
    bluebach said:

    ...Anyone ...?

    Well, I ran into this brick wall as well when I decided to try GPU mining out, triggering a switch from fast to full block mode on my node? I found it advantageous to switch to a SSD-based host early on, but then things were really bogged down when I started seeing mgasps=0.2xx or similar in low blockcount low transaction chain segments. It seems there may be some hope, depending on what you're dealing with. Are you at that slowdown at 2283397 and the subsequent clearing contract around 2675000?

    see also: https://ethereum.stackexchange.com/questions/9883/why-is-my-node-synchronization-stuck-extremely-slow-at-block-2-306-843
    todd said:

    It requires SSD with space available - presently 102Gb.

    ~/.ethereum$ du -skh geth
    130G	geth
    That storage usage was spotted at 2706667 on the full block mode sync, for what it's worth.
    Post edited by chroma on
  • vnikvnik Member Posts: 2
    I have the same issue. Using geth 1.7.2-stable-1db4ecdc. I've been syncing the blockchain for weeks now on a 100MBps connection. du -sh ~/.ethereum/geth is 162GB at the moment and I'm still at block 4574831. I'm running geth with --cache=2048 and don't think I'll ever be able to catch up to the current block. It got to the point where geth is barely syncing.
  • chromachroma Member Posts: 8
    edited January 2018
    Yes, I'm also using geth 1.7.2-stable-1db4ecdc as bundled with Mist-0.9.3 for Linux AMD64. ~/.ethereum/geth/chaindata just grows and grows. I'm at full block 4336000 now, geth's chaindata has risen to about 455GB spread across 291780 individual .ldb files. It's exceeded my available contiguous SSD storage volume, but fortunately it responds well enough to having relatively static old individual .ldb files symlink'd off to slower storage (e.g. HDD or even HDD-based NAS).

    Best of luck getting caught up to the present chain state! I'm not sure why nobody else has been talking about this remarkable storage demand, when most of the discussion I can find online suggests it "should" take up less than 128GB. This thing makes Bitcoin's full blockchain (~165GB) look tiny in comparison. If it's just something I'm doing wrong somehow, hopefully I can figure that out or someone here can point out my error.

    Well, I'll keep at it because I do believe in the open, decentralizated, pro-social, pro-community world that I understand the Ethereum ecosystem aims to champion and promote. Let's build a better world, together, and then inhabit it.
  • chromachroma Member Posts: 8
    edited January 2018
    I'm also running geth 1.7.2-stable-1db4ecdc as was bundled with the Linux AMD64 build of Mist-0.9.3. By about the point at which it has sync'd to full block 4336000, ~/.ethereum/geth/chaindata has grown to over 450GB spread acros over 291782 individual .ldb files.

    It seems like a quite remarkable level of storage, much larger than Bitcoin (which is presently ~165GB), and it leaves me wondering if I've made some error that it is nobody else has been mentioning half-TB storage needs for a geth-based Ethereum node in 2018.

    Fortunately, geth does seem to respond well to having individual .ldb files symlink'd off to slower (e.g.: HDD or HDD-backed NAS) storage, so you don't actually require a 512GB+ SSD to get through this at > 1.0 mgasps.

    Well, it's a bit painful but I do earnestly believe in the decentralized, pro-community, pro-social world that I understand the Ethereum ecosystem can help champion and promote. Let's build a better world together, and then inhabit it and flourish.
    Post edited by chroma on
  • chromachroma Member Posts: 8
    edited January 2018
    I've decided to start over, as it seems to have gotten stuck on "gas limit exceeded" transactions around there with no sign of progress and far far far too much disk usage. Maybe this time my Ethereum node will behave more like others describe it ought to.
  • HorsetoothlessHorsetoothless Member Posts: 4
    Ditto problem stuck at 4370001 blocks for weeks.
    state entries continue and data builds up.

    keep getting many invalid hash chains from multiple peers that all have the same invalid difficulty from same want difficulty
    Have funds in wallet from mining that need access to, Please advise been started over clean 0 blocks geth --fast --cache 2048
Sign In or Register to comment.