Right now my setup is as follows for boot:
PXE Boot as only option in BIOS
Wake on Power at PSU (autorestart)
USB thumb drive installed as HDD
Normal operation, the miner PXE boots. PXE tells the miner to that I havent pushed a new update, it boots from the image on the USB disk. If I push a new image, PXE reimages the disk then boots.
I have found a couple times, that when I un plug or plug in the USB drive the USB drive will become priority number one, over PXE boot, preventing me to utilize PXE. So I disabled boot disks entirely and rely fully on the PXE boot environment to tell it to boot from the disk.
Sometimes, when I boot, I can see it hang at the boot screen and the USB is being accessed (red flashing light on usb disk) after about 3 minutes it will sometimes go to PXE sometimes not. A shutdown remove the usb and plug back in normally resolves the issue (weird)
Also, more often then not, about every 3rd or 4th boot the filesystem seems to get corrupted and it hangs when PXE trys to have it boot from the disk (it keeps trying to boot from disk, but since the drive is screwed up it is stuck in a loop). I have to push a new update to the drive, reimaging it, that then seems to fix the issue, until it happens again.
My normal process for a stuck miner is to power down at the PDU socket, unfortunately it seems a hard shut down takes out 20% of my miners and I must do the above to resolve (weird).
I have the miners restart every 12 hours via crontab, and have even seen issues when it is restarting this way.
I am running several pro btc boards and wondered if any one else was running into these issues. I have a large enough operation that this isnt dependent on one rig, or one specific setup. As far as I can tell it is random. It may have something to do with the power down sequence but cant tell. Didnt know if any one else had experience with PXE booting and set up. I am starting to question weather or not my USB drives are bad or what is going on, I have never had this many corrupt filesystems or boot problems before, so any advice appreciated.