Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X...XX.......X...........] duration: 455.498566ms FAIL (2025-07-28 00:31:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.548803ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [..................................XXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031474286s FAIL (2025-07-28 00:31:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 4219 , piece_id=OHTAVZ76DTX2W7Q7MELTTADC6JQSJYXXVSZTOUZG5NVDZGTPK55Q, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.234052229 (26h38m16.014985615s) [noise=true , address=104.36.82.55:4227 , piece_id=H626XCVPCAGZJDRLDUNXNGN5UA4LTEFMJJLSZOY4BW5HJD5M3XUA, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.234836479 (26h38m16.015769871s) [noise=true , address=104.36.82.11:4131 , piece_id=RFGWMFFYM7LC4VPN2AZZJIUOP6DS6O6QSACX62HJLZI4TPVF2UAA, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.245807360 (26h38m16.02674075s) [noise=true , address=79.127.205.233:4656 , piece_id=GUF73BFGJLB7HFABZWEMVZV7JTNWAL4I73CCGFPFXZ5AHK3AKH4Q, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.246514014 (26h38m16.027447402s) [noise=true , address=104.36.82.56:4250 , piece_id=4Y6EYBR4KHN5GOCI6RDTI5GOJSNDWZZSZRVZINQJKSJIGCC5RKFQ, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.247043495 (26h38m16.027976882s) [noise=true , address=104.36.82.3:4779 , piece_id=TEA5SWPYBZ7YL3XPOZ5SR6WFFAZ3KSZHTZ47Y4HBX3ULBRRQ5YFA, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.247532705 (26h38m16.028466087s) [noise=true , address=104.36.82.44:4885 , piece_id=MHRN7HGC4CI2NKR6ALUN6IN7MISUYA4S5FB25PO23CJKAUDO3XGQ, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.248051611 (26h38m16.028984989s) [noise=true , address=104.36.82.21:4502 , piece_id=2VCSVECG3LVMYGMGR2BGF6ZN2L4GEMBFEXYHQB5BGAWWT2632BMQ, seg_pos={0 0}, upload=25422]: Uploading piece... Jul 28 00:30:00.316947193 (26h38m16.097880572s) [upload=25422]: Next returned nil segment Jul 28 00:30:00.316971107 (26h38m16.097904475s) [upload=25422]: Waiting for error group managing segments... Jul 28 00:30:00.479145027 (26h38m16.260078392s) [noise=true , address=45.91.116.9:4280 , piece_id=XUQ5MKVKS2QY4VSUZQGD3ABF5FBGNAEBR5A5W5WVZZHXDUDXNQJA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: <nil> Jul 28 00:30:00.830123366 (26h38m16.611056746s) [noise=true , address=79.127.205.233:4656 , piece_id=GUF73BFGJLB7HFABZWEMVZV7JTNWAL4I73CCGFPFXZ5AHK3AKH4Q, seg_pos={0 0}, upload=25422]: Done uploading piece. err: <nil> Jul 28 00:31:00.152786573 (26h39m15.933719959s) [noise=true , address=104.36.82.11:4131 , piece_id=RFGWMFFYM7LC4VPN2AZZJIUOP6DS6O6QSACX62HJLZI4TPVF2UAA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 159.223.143.39:49316->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.152872692 (26h39m15.933806057s) [noise=true , address=104.36.82.56:4250 , piece_id=4Y6EYBR4KHN5GOCI6RDTI5GOJSNDWZZSZRVZINQJKSJIGCC5RKFQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:12oHdA7jYDKSYwWLB975bJ3Fu27xjfZDC9utpnQpszmYjVz2CdD, address:104.36.82.56:4250): protocol: rpc: dial tcp 104.36.82.56:4250: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.152900978 (26h39m15.933834357s) [noise=true , address=104.36.82.7:4919 , piece_id=FO5CTHNDFPG6XELSLOKIE7GNVW7HNQ26UZSZYAUAEPT3CGAAXRHA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w, address:104.36.82.7:4919): protocol: write tcp 159.223.143.39:40626->104.36.82.7:4919: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.152947297 (26h39m15.933880676s) [noise=true , address=104.36.82.44:4885 , piece_id=MHRN7HGC4CI2NKR6ALUN6IN7MISUYA4S5FB25PO23CJKAUDO3XGQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: write tcp 159.223.143.39:36934->104.36.82.44:4885: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.152992066 (26h39m15.933925444s) [noise=true , address=104.36.82.52:4111 , piece_id=QLFU32JSUUJLH2Z77XGX7BCCZHPZ7HODSI22VFM3HFPOKSCVBPLQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: rpc: dial tcp 104.36.82.52:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153013223 (26h39m15.933946604s) [noise=true , address=104.36.82.21:4502 , piece_id=2VCSVECG3LVMYGMGR2BGF6ZN2L4GEMBFEXYHQB5BGAWWT2632BMQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:1T5ngNfyr72hJCDpsW6eJZq2YZoJphNYBE9gERs1qB6UkMxrHR, address:104.36.82.21:4502): protocol: write tcp 159.223.143.39:49746->104.36.82.21:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153075903 (26h39m15.934009281s) [noise=true , address=104.36.82.47:4520 , piece_id=B3EE5LPLI7DNXI33NLRCSR2XXCQG5DT67PPNN7TXFTEKCX3AG23Q, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: write tcp 159.223.143.39:60410->104.36.82.47:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153130960 (26h39m15.934064339s) [noise=true , address=104.36.82.6:4908 , piece_id=QYJ4GDIFGBSNGQYPGZ73ALPOTBJMUEHTI3DYFC773VFNNCUCTWUA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:104.36.82.6:4908): protocol: write tcp 159.223.143.39:32888->104.36.82.6:4908: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153165257 (26h39m15.934098638s) [noise=true , address=104.36.82.12:4152 , piece_id=ONQFMXCWAN2OB4KIDJ2P33KETAA6SBVPGH56EBQE7OVWYJI4BSEQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: write tcp 159.223.143.39:58278->104.36.82.12:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153191516 (26h39m15.934124895s) [noise=true , address=104.36.82.13:4219 , piece_id=OHTAVZ76DTX2W7Q7MELTTADC6JQSJYXXVSZTOUZG5NVDZGTPK55Q, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: write tcp 159.223.143.39:34184->104.36.82.13:4219: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153241290 (26h39m15.934174669s) [noise=true , address=104.36.82.56:4230 , piece_id=ZX7XUQBPQAO4FLYHGHFUAYY6D7TLDLVAR6WBEEEISAIJW4Z5VEPA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153251681 (26h39m15.934185044s) [noise=true , address=104.36.82.55:4227 , piece_id=H626XCVPCAGZJDRLDUNXNGN5UA4LTEFMJJLSZOY4BW5HJD5M3XUA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: rpc: dial tcp 104.36.82.55:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153296245 (26h39m15.934229608s) [noise=true , address=104.36.82.3:4779 , piece_id=TEA5SWPYBZ7YL3XPOZ5SR6WFFAZ3KSZHTZ47Y4HBX3ULBRRQ5YFA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:12DHKWSFHcN6FYabUR9Q8vpe7duLcyZmEPcDmqvvcQHDGm4bpvu, address:104.36.82.3:4779): protocol: rpc: dial tcp 104.36.82.3:4779: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153334065 (26h39m15.934267446s) [noise=true , address=104.36.82.15:4285 , piece_id=2WNIRPQ5QNCWZ2IIL6H6WDVLGEZD2JTVLEK4F7L47JKWB53Z2VMQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: write tcp 159.223.143.39:44516->104.36.82.15:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153343845 (26h39m15.934277207s) [noise=true , address=104.36.82.41:4782 , piece_id=5WGUMNZ5EHN7RJ3RY2EJ4IGJ5SL7XUH4ENVY6NVLJOYNYR7G4GTA, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:1vst2WW2D3NGbxPDDkjFQPxuwX3joAY77EdWCGXxU3H1NVGgcr, address:104.36.82.41:4782): protocol: rpc: dial tcp 104.36.82.41:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153385991 (26h39m15.934319358s) [noise=true , address=104.36.82.49:4987 , piece_id=53VBWHS3NZ23PXV4LJ2ICPP63DJH2Y2ST5CUG7K7YA37B62CGDOQ, seg_pos={0 0}, upload=25422]: Done uploading piece. err: ecclient: upload failed (node:1oZxiaBQD1grLu8d1HyERweeuXNXc1aBgyAoe1B86gqm6ypvBC, address:104.36.82.49:4987): protocol: rpc: dial tcp 104.36.82.49:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:31:00.153406581 (26h39m15.934339969s) [seg_pos={0 0}, upload=25422]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:31:00.180081557 (26h39m15.961014925s) [upload=25422]: Done uploading segments... Jul 28 00:31:00.180113027 (26h39m15.961046414s) [upload=25422]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.03149576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 210.186029ms FAIL (2025-07-28 00:30:36 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 210.300842ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X........X.............XX..] duration: 598.615326ms FAIL (2025-07-28 00:30:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606835386 (26h36m29.387768763s) [noise=true , address=104.36.82.16:4295 , piece_id=HP6LCPYG7KSE4YG6DWE2CLJD6HQUC6MYORSR2EXIRINIEMS32UXA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:35574->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606862776 (26h36m29.387796142s) [seg_pos={0 0}, upload=25387]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:28:13.633228465 (26h36m29.414161847s) [upload=25387]: Done uploading segments... Jul 28 00:28:13.633293380 (26h36m29.414226768s) [upload=25387]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 28 00:28:47.754303546 (26h37m3.535236926s) [upload=25403]: Starting upload Jul 28 00:28:47.754391189 (26h37m3.535324573s) [upload=25403]: Done starting upload Jul 28 00:28:47.754509086 (26h37m3.535442469s) [upload=25403]: Uploading segments... Jul 28 00:28:47.754775660 (26h37m3.535709026s) [upload=25403]: Got next segment. Inline: false Jul 28 00:28:47.754820858 (26h37m3.535754234s) [upload=25403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:28:47.754839152 (26h37m3.535772529s) [upload=25403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:28:47.780856736 (26h37m3.561790111s) [seg_pos={0 0}, upload=25403]: Begin upload segment... Jul 28 00:28:47.780922215 (26h37m3.561855581s) [seg_pos={0 0}, upload=25403]: Done begin upload segment. Jul 28 00:28:47.780976265 (26h37m3.561909655s) [noise=true , address=77.90.31.12:4589 , piece_id=B6VMSU6BDMIPJEJDV2ZCLKO4IE3GYN7U2MID4WNIHLNKSJRYKMJQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.780995574 (26h37m3.561928958s) [noise=true , address=45.91.116.6:4975 , piece_id=R7MAH6C366W6SBPA5WY6XVZOZHHZDXEK3IZTLRDOYU2SQIOTNMZA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781473500 (26h37m3.562406898s) [noise=true , address=45.91.116.4:4940 , piece_id=PB5YSEPHFXZ2E6K2IU4JVF2FHMY62PHAZJLXQB7EQEICHLTIDFDQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781605871 (26h37m3.562539258s) [noise=true , address=45.91.116.14:4190 , piece_id=AVWLHMUECVTX3WHIXMAVCDVEXMBZCHAP52J4WFO3LANP6QN6K3UQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781924118 (26h37m3.562857506s) [noise=true , address=45.91.116.10:4083 , piece_id=PDDTUEV25HYE2UMTRHNTYJI7OQFQQONVWKNYHYSGJQDRIKHXP3WA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782097161 (26h37m3.563030542s) [noise=true , address=45.91.116.5:4299 , piece_id=X45LWIN5BA6JP6THTGSSK7HAEBUIX6SJEUNTF7HFOHIMGNKEZEDQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782522027 (26h37m3.563455414s) [noise=true , address=77.90.31.21:4879 , piece_id=MRFNQQQKZVXDHXMCHKK2VJY3ZSPBVYDKR2FIJXNLN6LXBGIWD7AA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782654938 (26h37m3.56358831s) [noise=true , address=104.36.82.4:4801 , piece_id=5SXNBPEZ66VMLJUIF5MNP2KJ7BECOCW6ZHXLO5LHHIAEPLT4JPIQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782969242 (26h37m3.563902653s) [noise=true , address=77.90.31.15:4698 , piece_id=XSNGZ7OX2SHC43FULTNWE6RF34KXPH2KKPPC3ZRCE37JWUFGI6JQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.783417970 (26h37m3.564351359s) [noise=true , address=104.36.82.2:4730 , piece_id=FN2HDJGK4ETSBBQTFOA4AHCKH77QVPXC7HXCJMIRPFEGJVHAZFMA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.783845544 (26h37m3.564778935s) [noise=true , address=45.91.116.5:4298 , piece_id=T2AUFTQQ2DHLJ5QJBKUSJZTDXKQXUMQQS4VS22IVOCIFNZOXU7JA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.784394018 (26h37m3.565327408s) [noise=true , address=104.36.82.12:4151 , piece_id=FVZITDBMTDRRLKJGPNZTK5Y5BREZRV3OBFGTACPR4Y5BCNJJLHBA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.785076081 (26h37m3.566009451s) [noise=true , address=77.90.31.16:4735 , piece_id=Y76C2JDM3UDUH5LH3F32TKC3ML6CLJJRZXKTUAVITPTTCKW6RC4A, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.785949963 (26h37m3.566883351s) [noise=true , address=45.91.116.12:4225 , piece_id=FSVVTJRL67Y45W4L5RDQH7XAOSRLMOJ7QDRV3GL2AULNZLKXFRXA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.786795975 (26h37m3.567729359s) [noise=true , address=45.91.116.13:4153 , piece_id=VRZRTXEUPMHZ5SGXH6OG2Z4LHH5GM2UEI6357WCTHGVNIZY5FRIQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.787590649 (26h37m3.568524036s) [noise=true , address=104.36.82.5:4838 , piece_id=FQKT6UGQIJJP7IPEB7EX4MSW5AL27NEGW6KWTLSGL372FG7Y5A3Q, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.957449281 (26h37m3.738382662s) [upload=25403]: Next returned nil segment Jul 28 00:28:47.957468373 (26h37m3.738401763s) [upload=25403]: Waiting for error group managing segments... Jul 28 00:28:47.962742193 (26h37m3.743675577s) [noise=true , address=45.91.116.5:4299 , piece_id=X45LWIN5BA6JP6THTGSSK7HAEBUIX6SJEUNTF7HFOHIMGNKEZEDQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.963488603 (26h37m3.744421987s) [noise=true , address=45.91.116.4:4940 , piece_id=PB5YSEPHFXZ2E6K2IU4JVF2FHMY62PHAZJLXQB7EQEICHLTIDFDQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.964153862 (26h37m3.745087248s) [noise=true , address=45.91.116.14:4190 , piece_id=AVWLHMUECVTX3WHIXMAVCDVEXMBZCHAP52J4WFO3LANP6QN6K3UQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.965037990 (26h37m3.745971376s) [noise=true , address=45.91.116.5:4298 , piece_id=T2AUFTQQ2DHLJ5QJBKUSJZTDXKQXUMQQS4VS22IVOCIFNZOXU7JA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.965759371 (26h37m3.746692759s) [noise=true , address=45.91.116.13:4153 , piece_id=VRZRTXEUPMHZ5SGXH6OG2Z4LHH5GM2UEI6357WCTHGVNIZY5FRIQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.966447263 (26h37m3.747380644s) [noise=true , address=45.91.116.10:4083 , piece_id=PDDTUEV25HYE2UMTRHNTYJI7OQFQQONVWKNYHYSGJQDRIKHXP3WA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.967731901 (26h37m3.748665284s) [noise=true , address=77.90.31.21:4879 , piece_id=MRFNQQQKZVXDHXMCHKK2VJY3ZSPBVYDKR2FIJXNLN6LXBGIWD7AA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.969381419 (26h37m3.750314801s) [noise=true , address=77.90.31.12:4589 , piece_id=B6VMSU6BDMIPJEJDV2ZCLKO4IE3GYN7U2MID4WNIHLNKSJRYKMJQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.970091242 (26h37m3.751024627s) [noise=true , address=77.90.31.15:4698 , piece_id=XSNGZ7OX2SHC43FULTNWE6RF34KXPH2KKPPC3ZRCE37JWUFGI6JQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.970418569 (26h37m3.751351953s) [noise=true , address=45.91.116.12:4225 , piece_id=FSVVTJRL67Y45W4L5RDQH7XAOSRLMOJ7QDRV3GL2AULNZLKXFRXA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.971157743 (26h37m3.752091123s) [noise=true , address=45.91.116.6:4975 , piece_id=R7MAH6C366W6SBPA5WY6XVZOZHHZDXEK3IZTLRDOYU2SQIOTNMZA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.991275219 (26h37m3.772208597s) [noise=true , address=77.90.31.16:4735 , piece_id=Y76C2JDM3UDUH5LH3F32TKC3ML6CLJJRZXKTUAVITPTTCKW6RC4A, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:29:17.719322026 (26h37m33.500255412s) [noise=true , address=104.36.82.4:4801 , piece_id=5SXNBPEZ66VMLJUIF5MNP2KJ7BECOCW6ZHXLO5LHHIAEPLT4JPIQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719426812 (26h37m33.500360193s) [noise=true , address=104.36.82.5:4838 , piece_id=FQKT6UGQIJJP7IPEB7EX4MSW5AL27NEGW6KWTLSGL372FG7Y5A3Q, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:53884->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719538823 (26h37m33.50047219s) [noise=true , address=104.36.82.2:4730 , piece_id=FN2HDJGK4ETSBBQTFOA4AHCKH77QVPXC7HXCJMIRPFEGJVHAZFMA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 28 00:29:17.719626819 (26h37m33.500560197s) [noise=true , address=104.36.82.12:4151 , piece_id=FVZITDBMTDRRLKJGPNZTK5Y5BREZRV3OBFGTACPR4Y5BCNJJLHBA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719657452 (26h37m33.500590836s) [seg_pos={0 0}, upload=25403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:17.745058047 (26h37m33.525991414s) [upload=25403]: Done uploading segments... Jul 28 00:29:17.745079722 (26h37m33.526013086s) [upload=25403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 598.643293ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X...XX.......X..........] duration: 499.152492ms FAIL (2025-07-28 00:30:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.183713ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [...................................XXX.XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03781522s FAIL (2025-07-28 00:30:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 4.36.82.20:4448 , piece_id=DITI4FIVMABVXJ3ZVL75MBQZFLLL4EEI2M3QI2UQCJGE6N57APFQ, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.121188890 (26h37m15.902122271s) [noise=true , address=104.36.82.9:4037 , piece_id=MGI5ZBEKRMVBLACXE7AGVNY6S5UDYMIVGTVQKUEVJT35GNLQSXGQ, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.121479559 (26h37m15.902412937s) [noise=true , address=104.36.82.41:4764 , piece_id=K4E7EAMNIFN2BQTRFJEUFIKHQEUI7KKKUI2CWTN23WS7BKCFBKOA, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.121775270 (26h37m15.902708651s) [noise=true , address=104.36.82.13:4201 , piece_id=HNVS7GAWNCKVBEOM2PLTVHJPF4CS6ZNL7XOU6ABQQWGT3PNKX7MA, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.122075693 (26h37m15.903009073s) [noise=true , address=104.36.82.14:4225 , piece_id=KJ6HCWAFVNOT4KKGYR3FLHVYVDNAUTYCNZ6ZLUDIRQENMXESN35A, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.122396630 (26h37m15.903330008s) [noise=true , address=104.36.82.55:4212 , piece_id=2WLU45HJ7OO64AKASU2NOZ2Y53EL5O7ODDQU64KXGYS3GIAPIIGQ, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.122716814 (26h37m15.903650195s) [noise=true , address=104.36.82.15:4274 , piece_id=HR5DTFSXJL6STM3RCFKHBN7RN3PE5XCR4G4JTMP6UVXIKZ3RT56Q, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.123161181 (26h37m15.90409457s) [noise=true , address=104.36.82.12:4163 , piece_id=3UBAFVNQ2EBE66565IECJP2EELWLBMW3UNAIMWGOKHHFOUQ5WWFA, seg_pos={0 0}, upload=25406]: Uploading piece... Jul 28 00:29:00.335833222 (26h37m16.116766586s) [upload=25406]: Next returned nil segment Jul 28 00:29:00.335857488 (26h37m16.116790882s) [upload=25406]: Waiting for error group managing segments... Jul 28 00:29:03.320620974 (26h37m19.101554336s) [noise=true , address=79.127.201.215:4355 , piece_id=JOJNU2PAZ52VEQGEZVF447Q72JXJMTZN2XSHX43NQEQS6HIJFG6Q, seg_pos={0 0}, upload=25406]: Done uploading piece. err: <nil> Jul 28 00:30:00.046256009 (26h38m15.827189398s) [noise=true , address=104.36.82.5:4854 , piece_id=WTBGCOYZPSCYJBTRK24HUOPAHTW3Z54QXTSBYNDQTZPTUDQZYUXA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy, address:104.36.82.5:4854): protocol: rpc: dial tcp 104.36.82.5:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046358313 (26h38m15.827291698s) [noise=true , address=104.36.82.45:4907 , piece_id=AEDLF6RRRSF4SJUUIDZ7IIY73S4UTV534O24KTXDTBDMU4ZSHWPA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:104.36.82.45:4907): protocol: rpc: dial tcp 104.36.82.45:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046433518 (26h38m15.827366901s) [noise=true , address=104.36.82.20:4448 , piece_id=DITI4FIVMABVXJ3ZVL75MBQZFLLL4EEI2M3QI2UQCJGE6N57APFQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: rpc: dial tcp 104.36.82.20:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046508379 (26h38m15.827441759s) [noise=true , address=104.36.82.41:4764 , piece_id=K4E7EAMNIFN2BQTRFJEUFIKHQEUI7KKKUI2CWTN23WS7BKCFBKOA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: rpc: dial tcp 104.36.82.41:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046578670 (26h38m15.827512034s) [noise=true , address=104.36.82.14:4225 , piece_id=KJ6HCWAFVNOT4KKGYR3FLHVYVDNAUTYCNZ6ZLUDIRQENMXESN35A, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:104.36.82.14:4225): protocol: rpc: dial tcp 104.36.82.14:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046642740 (26h38m15.82757612s) [noise=true , address=104.36.82.15:4274 , piece_id=HR5DTFSXJL6STM3RCFKHBN7RN3PE5XCR4G4JTMP6UVXIKZ3RT56Q, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12nZ58ntQujpYk3UVGt4rD6xssGW3RxmvgM84a4Ka8BCRowuaim, address:104.36.82.15:4274): protocol: rpc: dial tcp 104.36.82.15:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046708855 (26h38m15.827642218s) [noise=true , address=104.36.82.52:4106 , piece_id=B5UA5F4RRZIB2P7DOICVSKK7TMW2QYQGIK65ACE43LQGWUD6OWOQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046775735 (26h38m15.827709115s) [noise=true , address=104.36.82.42:4815 , piece_id=3NZNPXOV2T23FXOPBPSZFIO7AGQY3OLMDFADSS7LTTY3PGPXQQAQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:1F59CEsZMSXL8DibXEqG6XQKirqjEfYW8UhugnFCPnUEUWLHDX, address:104.36.82.42:4815): protocol: write tcp 159.223.143.39:48688->104.36.82.42:4815: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046841728 (26h38m15.827775108s) [noise=true , address=104.36.82.3:4781 , piece_id=2FMUPAVAFA3CYD4UR7TKLLAZTDDCKXOANZYB5G3YQUY6FRAC5GHA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:1Y6iagayTdoSQtXrAdeBrvrck8qMNRAmLogi5jUG8bcLrBEswn, address:104.36.82.3:4781): protocol: rpc: dial tcp 104.36.82.3:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.046923683 (26h38m15.827857065s) [noise=true , address=104.36.82.9:4037 , piece_id=MGI5ZBEKRMVBLACXE7AGVNY6S5UDYMIVGTVQKUEVJT35GNLQSXGQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:1z623XGBbui7vxQHfszNECGnEXw5VBnxmh5VyvwX5RsmXL9SNz, address:104.36.82.9:4037): protocol: rpc: dial tcp 104.36.82.9:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047007905 (26h38m15.827941283s) [noise=true , address=104.36.82.55:4212 , piece_id=2WLU45HJ7OO64AKASU2NOZ2Y53EL5O7ODDQU64KXGYS3GIAPIIGQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: rpc: dial tcp 104.36.82.55:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047123169 (26h38m15.828056548s) [noise=true , address=104.36.82.12:4163 , piece_id=3UBAFVNQ2EBE66565IECJP2EELWLBMW3UNAIMWGOKHHFOUQ5WWFA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:104.36.82.12:4163): protocol: rpc: dial tcp 104.36.82.12:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047150660 (26h38m15.828084028s) [noise=true , address=104.36.82.51:4067 , piece_id=EY4FCJYJ2UCDHLLCJBJN7KKFGI43NODNKEBHDWNXMOKGJCCTUWYQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq, address:104.36.82.51:4067): protocol: write tcp 159.223.143.39:33202->104.36.82.51:4067: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047169050 (26h38m15.828102429s) [noise=true , address=104.36.82.53:4124 , piece_id=U7QZ3HW3V7IHABPS2TEM4FEWBFA67JBUT3EXBARKMJJD7GGDNNJQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047261144 (26h38m15.828194507s) [noise=true , address=104.36.82.13:4201 , piece_id=HNVS7GAWNCKVBEOM2PLTVHJPF4CS6ZNL7XOU6ABQQWGT3PNKX7MA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:104.36.82.13:4201): protocol: rpc: dial tcp 104.36.82.13:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047269238 (26h38m15.828202601s) [noise=true , address=104.36.82.49:4998 , piece_id=4UPHXJWFV4YJHISNRDZ5BZ7KJN3BT6KQMB6BIOQ2FPPAZITFCJCQ, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:104.36.82.49:4998): protocol: rpc: dial tcp 104.36.82.49:4998: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047347807 (26h38m15.828281189s) [noise=true , address=104.36.82.34:4570 , piece_id=HR2FECG7OBF4PT4QUCBXHI7ZPTVKEZ526YPXYV6HS6M3YVRQFZGA, seg_pos={0 0}, upload=25406]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: write tcp 159.223.143.39:38884->104.36.82.34:4570: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:30:00.047429182 (26h38m15.828362573s) [seg_pos={0 0}, upload=25406]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:30:00.081163089 (26h38m15.862096466s) [upload=25406]: Done uploading segments... Jul 28 00:30:00.081194340 (26h38m15.86212773s) [upload=25406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.037847992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X........X.............XX.] duration: 455.207056ms FAIL (2025-07-28 00:29:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606835386 (26h36m29.387768763s) [noise=true , address=104.36.82.16:4295 , piece_id=HP6LCPYG7KSE4YG6DWE2CLJD6HQUC6MYORSR2EXIRINIEMS32UXA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:35574->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606862776 (26h36m29.387796142s) [seg_pos={0 0}, upload=25387]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:28:13.633228465 (26h36m29.414161847s) [upload=25387]: Done uploading segments... Jul 28 00:28:13.633293380 (26h36m29.414226768s) [upload=25387]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 28 00:28:47.754303546 (26h37m3.535236926s) [upload=25403]: Starting upload Jul 28 00:28:47.754391189 (26h37m3.535324573s) [upload=25403]: Done starting upload Jul 28 00:28:47.754509086 (26h37m3.535442469s) [upload=25403]: Uploading segments... Jul 28 00:28:47.754775660 (26h37m3.535709026s) [upload=25403]: Got next segment. Inline: false Jul 28 00:28:47.754820858 (26h37m3.535754234s) [upload=25403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:28:47.754839152 (26h37m3.535772529s) [upload=25403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:28:47.780856736 (26h37m3.561790111s) [seg_pos={0 0}, upload=25403]: Begin upload segment... Jul 28 00:28:47.780922215 (26h37m3.561855581s) [seg_pos={0 0}, upload=25403]: Done begin upload segment. Jul 28 00:28:47.780976265 (26h37m3.561909655s) [noise=true , address=77.90.31.12:4589 , piece_id=B6VMSU6BDMIPJEJDV2ZCLKO4IE3GYN7U2MID4WNIHLNKSJRYKMJQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.780995574 (26h37m3.561928958s) [noise=true , address=45.91.116.6:4975 , piece_id=R7MAH6C366W6SBPA5WY6XVZOZHHZDXEK3IZTLRDOYU2SQIOTNMZA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781473500 (26h37m3.562406898s) [noise=true , address=45.91.116.4:4940 , piece_id=PB5YSEPHFXZ2E6K2IU4JVF2FHMY62PHAZJLXQB7EQEICHLTIDFDQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781605871 (26h37m3.562539258s) [noise=true , address=45.91.116.14:4190 , piece_id=AVWLHMUECVTX3WHIXMAVCDVEXMBZCHAP52J4WFO3LANP6QN6K3UQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781924118 (26h37m3.562857506s) [noise=true , address=45.91.116.10:4083 , piece_id=PDDTUEV25HYE2UMTRHNTYJI7OQFQQONVWKNYHYSGJQDRIKHXP3WA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782097161 (26h37m3.563030542s) [noise=true , address=45.91.116.5:4299 , piece_id=X45LWIN5BA6JP6THTGSSK7HAEBUIX6SJEUNTF7HFOHIMGNKEZEDQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782522027 (26h37m3.563455414s) [noise=true , address=77.90.31.21:4879 , piece_id=MRFNQQQKZVXDHXMCHKK2VJY3ZSPBVYDKR2FIJXNLN6LXBGIWD7AA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782654938 (26h37m3.56358831s) [noise=true , address=104.36.82.4:4801 , piece_id=5SXNBPEZ66VMLJUIF5MNP2KJ7BECOCW6ZHXLO5LHHIAEPLT4JPIQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782969242 (26h37m3.563902653s) [noise=true , address=77.90.31.15:4698 , piece_id=XSNGZ7OX2SHC43FULTNWE6RF34KXPH2KKPPC3ZRCE37JWUFGI6JQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.783417970 (26h37m3.564351359s) [noise=true , address=104.36.82.2:4730 , piece_id=FN2HDJGK4ETSBBQTFOA4AHCKH77QVPXC7HXCJMIRPFEGJVHAZFMA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.783845544 (26h37m3.564778935s) [noise=true , address=45.91.116.5:4298 , piece_id=T2AUFTQQ2DHLJ5QJBKUSJZTDXKQXUMQQS4VS22IVOCIFNZOXU7JA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.784394018 (26h37m3.565327408s) [noise=true , address=104.36.82.12:4151 , piece_id=FVZITDBMTDRRLKJGPNZTK5Y5BREZRV3OBFGTACPR4Y5BCNJJLHBA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.785076081 (26h37m3.566009451s) [noise=true , address=77.90.31.16:4735 , piece_id=Y76C2JDM3UDUH5LH3F32TKC3ML6CLJJRZXKTUAVITPTTCKW6RC4A, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.785949963 (26h37m3.566883351s) [noise=true , address=45.91.116.12:4225 , piece_id=FSVVTJRL67Y45W4L5RDQH7XAOSRLMOJ7QDRV3GL2AULNZLKXFRXA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.786795975 (26h37m3.567729359s) [noise=true , address=45.91.116.13:4153 , piece_id=VRZRTXEUPMHZ5SGXH6OG2Z4LHH5GM2UEI6357WCTHGVNIZY5FRIQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.787590649 (26h37m3.568524036s) [noise=true , address=104.36.82.5:4838 , piece_id=FQKT6UGQIJJP7IPEB7EX4MSW5AL27NEGW6KWTLSGL372FG7Y5A3Q, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.957449281 (26h37m3.738382662s) [upload=25403]: Next returned nil segment Jul 28 00:28:47.957468373 (26h37m3.738401763s) [upload=25403]: Waiting for error group managing segments... Jul 28 00:28:47.962742193 (26h37m3.743675577s) [noise=true , address=45.91.116.5:4299 , piece_id=X45LWIN5BA6JP6THTGSSK7HAEBUIX6SJEUNTF7HFOHIMGNKEZEDQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.963488603 (26h37m3.744421987s) [noise=true , address=45.91.116.4:4940 , piece_id=PB5YSEPHFXZ2E6K2IU4JVF2FHMY62PHAZJLXQB7EQEICHLTIDFDQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.964153862 (26h37m3.745087248s) [noise=true , address=45.91.116.14:4190 , piece_id=AVWLHMUECVTX3WHIXMAVCDVEXMBZCHAP52J4WFO3LANP6QN6K3UQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.965037990 (26h37m3.745971376s) [noise=true , address=45.91.116.5:4298 , piece_id=T2AUFTQQ2DHLJ5QJBKUSJZTDXKQXUMQQS4VS22IVOCIFNZOXU7JA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.965759371 (26h37m3.746692759s) [noise=true , address=45.91.116.13:4153 , piece_id=VRZRTXEUPMHZ5SGXH6OG2Z4LHH5GM2UEI6357WCTHGVNIZY5FRIQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.966447263 (26h37m3.747380644s) [noise=true , address=45.91.116.10:4083 , piece_id=PDDTUEV25HYE2UMTRHNTYJI7OQFQQONVWKNYHYSGJQDRIKHXP3WA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.967731901 (26h37m3.748665284s) [noise=true , address=77.90.31.21:4879 , piece_id=MRFNQQQKZVXDHXMCHKK2VJY3ZSPBVYDKR2FIJXNLN6LXBGIWD7AA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.969381419 (26h37m3.750314801s) [noise=true , address=77.90.31.12:4589 , piece_id=B6VMSU6BDMIPJEJDV2ZCLKO4IE3GYN7U2MID4WNIHLNKSJRYKMJQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.970091242 (26h37m3.751024627s) [noise=true , address=77.90.31.15:4698 , piece_id=XSNGZ7OX2SHC43FULTNWE6RF34KXPH2KKPPC3ZRCE37JWUFGI6JQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.970418569 (26h37m3.751351953s) [noise=true , address=45.91.116.12:4225 , piece_id=FSVVTJRL67Y45W4L5RDQH7XAOSRLMOJ7QDRV3GL2AULNZLKXFRXA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.971157743 (26h37m3.752091123s) [noise=true , address=45.91.116.6:4975 , piece_id=R7MAH6C366W6SBPA5WY6XVZOZHHZDXEK3IZTLRDOYU2SQIOTNMZA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.991275219 (26h37m3.772208597s) [noise=true , address=77.90.31.16:4735 , piece_id=Y76C2JDM3UDUH5LH3F32TKC3ML6CLJJRZXKTUAVITPTTCKW6RC4A, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:29:17.719322026 (26h37m33.500255412s) [noise=true , address=104.36.82.4:4801 , piece_id=5SXNBPEZ66VMLJUIF5MNP2KJ7BECOCW6ZHXLO5LHHIAEPLT4JPIQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719426812 (26h37m33.500360193s) [noise=true , address=104.36.82.5:4838 , piece_id=FQKT6UGQIJJP7IPEB7EX4MSW5AL27NEGW6KWTLSGL372FG7Y5A3Q, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:53884->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719538823 (26h37m33.50047219s) [noise=true , address=104.36.82.2:4730 , piece_id=FN2HDJGK4ETSBBQTFOA4AHCKH77QVPXC7HXCJMIRPFEGJVHAZFMA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 28 00:29:17.719626819 (26h37m33.500560197s) [noise=true , address=104.36.82.12:4151 , piece_id=FVZITDBMTDRRLKJGPNZTK5Y5BREZRV3OBFGTACPR4Y5BCNJJLHBA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719657452 (26h37m33.500590836s) [seg_pos={0 0}, upload=25403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:17.745058047 (26h37m33.525991414s) [upload=25403]: Done uploading segments... Jul 28 00:29:17.745079722 (26h37m33.526013086s) [upload=25403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.238227ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 319.468063ms FAIL (2025-07-28 00:29:34 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 319.516701ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X...XX.......X.........] duration: 421.9986ms FAIL (2025-07-28 00:29:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.033819ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X........X.............XX] duration: 30.027055105s FAIL (2025-07-28 00:29:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606835386 (26h36m29.387768763s) [noise=true , address=104.36.82.16:4295 , piece_id=HP6LCPYG7KSE4YG6DWE2CLJD6HQUC6MYORSR2EXIRINIEMS32UXA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:35574->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606862776 (26h36m29.387796142s) [seg_pos={0 0}, upload=25387]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:28:13.633228465 (26h36m29.414161847s) [upload=25387]: Done uploading segments... Jul 28 00:28:13.633293380 (26h36m29.414226768s) [upload=25387]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 28 00:28:47.754303546 (26h37m3.535236926s) [upload=25403]: Starting upload Jul 28 00:28:47.754391189 (26h37m3.535324573s) [upload=25403]: Done starting upload Jul 28 00:28:47.754509086 (26h37m3.535442469s) [upload=25403]: Uploading segments... Jul 28 00:28:47.754775660 (26h37m3.535709026s) [upload=25403]: Got next segment. Inline: false Jul 28 00:28:47.754820858 (26h37m3.535754234s) [upload=25403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:28:47.754839152 (26h37m3.535772529s) [upload=25403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:28:47.780856736 (26h37m3.561790111s) [seg_pos={0 0}, upload=25403]: Begin upload segment... Jul 28 00:28:47.780922215 (26h37m3.561855581s) [seg_pos={0 0}, upload=25403]: Done begin upload segment. Jul 28 00:28:47.780976265 (26h37m3.561909655s) [noise=true , address=77.90.31.12:4589 , piece_id=B6VMSU6BDMIPJEJDV2ZCLKO4IE3GYN7U2MID4WNIHLNKSJRYKMJQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.780995574 (26h37m3.561928958s) [noise=true , address=45.91.116.6:4975 , piece_id=R7MAH6C366W6SBPA5WY6XVZOZHHZDXEK3IZTLRDOYU2SQIOTNMZA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781473500 (26h37m3.562406898s) [noise=true , address=45.91.116.4:4940 , piece_id=PB5YSEPHFXZ2E6K2IU4JVF2FHMY62PHAZJLXQB7EQEICHLTIDFDQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781605871 (26h37m3.562539258s) [noise=true , address=45.91.116.14:4190 , piece_id=AVWLHMUECVTX3WHIXMAVCDVEXMBZCHAP52J4WFO3LANP6QN6K3UQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.781924118 (26h37m3.562857506s) [noise=true , address=45.91.116.10:4083 , piece_id=PDDTUEV25HYE2UMTRHNTYJI7OQFQQONVWKNYHYSGJQDRIKHXP3WA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782097161 (26h37m3.563030542s) [noise=true , address=45.91.116.5:4299 , piece_id=X45LWIN5BA6JP6THTGSSK7HAEBUIX6SJEUNTF7HFOHIMGNKEZEDQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782522027 (26h37m3.563455414s) [noise=true , address=77.90.31.21:4879 , piece_id=MRFNQQQKZVXDHXMCHKK2VJY3ZSPBVYDKR2FIJXNLN6LXBGIWD7AA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782654938 (26h37m3.56358831s) [noise=true , address=104.36.82.4:4801 , piece_id=5SXNBPEZ66VMLJUIF5MNP2KJ7BECOCW6ZHXLO5LHHIAEPLT4JPIQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.782969242 (26h37m3.563902653s) [noise=true , address=77.90.31.15:4698 , piece_id=XSNGZ7OX2SHC43FULTNWE6RF34KXPH2KKPPC3ZRCE37JWUFGI6JQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.783417970 (26h37m3.564351359s) [noise=true , address=104.36.82.2:4730 , piece_id=FN2HDJGK4ETSBBQTFOA4AHCKH77QVPXC7HXCJMIRPFEGJVHAZFMA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.783845544 (26h37m3.564778935s) [noise=true , address=45.91.116.5:4298 , piece_id=T2AUFTQQ2DHLJ5QJBKUSJZTDXKQXUMQQS4VS22IVOCIFNZOXU7JA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.784394018 (26h37m3.565327408s) [noise=true , address=104.36.82.12:4151 , piece_id=FVZITDBMTDRRLKJGPNZTK5Y5BREZRV3OBFGTACPR4Y5BCNJJLHBA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.785076081 (26h37m3.566009451s) [noise=true , address=77.90.31.16:4735 , piece_id=Y76C2JDM3UDUH5LH3F32TKC3ML6CLJJRZXKTUAVITPTTCKW6RC4A, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.785949963 (26h37m3.566883351s) [noise=true , address=45.91.116.12:4225 , piece_id=FSVVTJRL67Y45W4L5RDQH7XAOSRLMOJ7QDRV3GL2AULNZLKXFRXA, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.786795975 (26h37m3.567729359s) [noise=true , address=45.91.116.13:4153 , piece_id=VRZRTXEUPMHZ5SGXH6OG2Z4LHH5GM2UEI6357WCTHGVNIZY5FRIQ, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.787590649 (26h37m3.568524036s) [noise=true , address=104.36.82.5:4838 , piece_id=FQKT6UGQIJJP7IPEB7EX4MSW5AL27NEGW6KWTLSGL372FG7Y5A3Q, seg_pos={0 0}, upload=25403]: Uploading piece... Jul 28 00:28:47.957449281 (26h37m3.738382662s) [upload=25403]: Next returned nil segment Jul 28 00:28:47.957468373 (26h37m3.738401763s) [upload=25403]: Waiting for error group managing segments... Jul 28 00:28:47.962742193 (26h37m3.743675577s) [noise=true , address=45.91.116.5:4299 , piece_id=X45LWIN5BA6JP6THTGSSK7HAEBUIX6SJEUNTF7HFOHIMGNKEZEDQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.963488603 (26h37m3.744421987s) [noise=true , address=45.91.116.4:4940 , piece_id=PB5YSEPHFXZ2E6K2IU4JVF2FHMY62PHAZJLXQB7EQEICHLTIDFDQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.964153862 (26h37m3.745087248s) [noise=true , address=45.91.116.14:4190 , piece_id=AVWLHMUECVTX3WHIXMAVCDVEXMBZCHAP52J4WFO3LANP6QN6K3UQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.965037990 (26h37m3.745971376s) [noise=true , address=45.91.116.5:4298 , piece_id=T2AUFTQQ2DHLJ5QJBKUSJZTDXKQXUMQQS4VS22IVOCIFNZOXU7JA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.965759371 (26h37m3.746692759s) [noise=true , address=45.91.116.13:4153 , piece_id=VRZRTXEUPMHZ5SGXH6OG2Z4LHH5GM2UEI6357WCTHGVNIZY5FRIQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.966447263 (26h37m3.747380644s) [noise=true , address=45.91.116.10:4083 , piece_id=PDDTUEV25HYE2UMTRHNTYJI7OQFQQONVWKNYHYSGJQDRIKHXP3WA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.967731901 (26h37m3.748665284s) [noise=true , address=77.90.31.21:4879 , piece_id=MRFNQQQKZVXDHXMCHKK2VJY3ZSPBVYDKR2FIJXNLN6LXBGIWD7AA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.969381419 (26h37m3.750314801s) [noise=true , address=77.90.31.12:4589 , piece_id=B6VMSU6BDMIPJEJDV2ZCLKO4IE3GYN7U2MID4WNIHLNKSJRYKMJQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.970091242 (26h37m3.751024627s) [noise=true , address=77.90.31.15:4698 , piece_id=XSNGZ7OX2SHC43FULTNWE6RF34KXPH2KKPPC3ZRCE37JWUFGI6JQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.970418569 (26h37m3.751351953s) [noise=true , address=45.91.116.12:4225 , piece_id=FSVVTJRL67Y45W4L5RDQH7XAOSRLMOJ7QDRV3GL2AULNZLKXFRXA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.971157743 (26h37m3.752091123s) [noise=true , address=45.91.116.6:4975 , piece_id=R7MAH6C366W6SBPA5WY6XVZOZHHZDXEK3IZTLRDOYU2SQIOTNMZA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:28:47.991275219 (26h37m3.772208597s) [noise=true , address=77.90.31.16:4735 , piece_id=Y76C2JDM3UDUH5LH3F32TKC3ML6CLJJRZXKTUAVITPTTCKW6RC4A, seg_pos={0 0}, upload=25403]: Done uploading piece. err: <nil> Jul 28 00:29:17.719322026 (26h37m33.500255412s) [noise=true , address=104.36.82.4:4801 , piece_id=5SXNBPEZ66VMLJUIF5MNP2KJ7BECOCW6ZHXLO5LHHIAEPLT4JPIQ, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719426812 (26h37m33.500360193s) [noise=true , address=104.36.82.5:4838 , piece_id=FQKT6UGQIJJP7IPEB7EX4MSW5AL27NEGW6KWTLSGL372FG7Y5A3Q, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:53884->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719538823 (26h37m33.50047219s) [noise=true , address=104.36.82.2:4730 , piece_id=FN2HDJGK4ETSBBQTFOA4AHCKH77QVPXC7HXCJMIRPFEGJVHAZFMA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 28 00:29:17.719626819 (26h37m33.500560197s) [noise=true , address=104.36.82.12:4151 , piece_id=FVZITDBMTDRRLKJGPNZTK5Y5BREZRV3OBFGTACPR4Y5BCNJJLHBA, seg_pos={0 0}, upload=25403]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:17.719657452 (26h37m33.500590836s) [seg_pos={0 0}, upload=25403]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:17.745058047 (26h37m33.525991414s) [upload=25403]: Done uploading segments... Jul 28 00:29:17.745079722 (26h37m33.526013086s) [upload=25403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027081816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 234.956313ms FAIL (2025-07-28 00:29:02 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 234.988822ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [....................................XXX.XXXXXXXXXXXXXXXXXXXX] duration: 1m0.036034523s FAIL (2025-07-28 00:28:59 UTC) ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== RS6TB5IW3OCU3I5IYCWVFPECOPFOG4QVJ253AYLA, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:27:59.279379465 (26h36m15.060312858s) [noise=true , address=104.36.82.44:4903 , piece_id=7PDRI2JXUOIIRMKZPNN3FFFYU7CU2O6D2GLTFL3DFXP7GZDY6C5A, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:27:59.283775328 (26h36m15.064708711s) [noise=true , address=104.36.82.15:4288 , piece_id=OLEBRCCOP3MGXZA34MFYSRQHIVOHBZFEMGF3HLFHGRFRF2XVONPA, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:27:59.284268725 (26h36m15.065202115s) [noise=true , address=104.36.82.20:4440 , piece_id=YK6VCCR66N2L37VP2I3AJK3LCCJVQ4IX524RFSHXBWU3S2I2HP3A, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:27:59.286104194 (26h36m15.067037573s) [noise=true , address=104.36.82.39:4756 , piece_id=ZWZ4ORCWAW6S63YSYJYA7CAZE7HSXAVHBRG74M336BZFT6PHN2GQ, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:27:59.286467396 (26h36m15.067400777s) [noise=true , address=104.36.82.41:4775 , piece_id=OLZV77VGFYFHY7N4COUPT4WZ6BO67Q6IGGLND3AHYCTZEU6FCOSA, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:27:59.286765120 (26h36m15.067698498s) [noise=true , address=104.36.82.56:4241 , piece_id=S4AWYAUXNB5SIGJYJPBNAWIN77IJHWSNFZCCXPQOBAR7AMMMPYKA, seg_pos={0 0}, upload=25392]: Uploading piece... Jul 28 00:28:59.115662140 (26h37m14.896595523s) [noise=true , address=104.36.82.15:4288 , piece_id=OLEBRCCOP3MGXZA34MFYSRQHIVOHBZFEMGF3HLFHGRFRF2XVONPA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:104.36.82.15:4288): protocol: rpc: dial tcp 104.36.82.15:4288: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.115754031 (26h37m14.896687411s) [noise=true , address=104.36.82.12:4162 , piece_id=A3NKLBOPPDKKSX3EX3XOD3CU7A6CKFEP3G56UM7UAASKW3P5IPAQ, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: rpc: dial tcp 104.36.82.12:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.115825137 (26h37m14.896758513s) [noise=true , address=104.36.82.4:4821 , piece_id=G3E5YC3QERRPC3DO2AD23TN4ICH2JVU6GSM56REX25MVV65UVF2Q, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: rpc: dial tcp 104.36.82.4:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116060361 (26h37m14.896993728s) [noise=true , address=104.36.82.53:4122 , piece_id=VEGWI76I2ERI2K7Y5JQ5UKBWG2JVP4EDLIFHKGEHI6BWU55KKCRA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 159.223.143.39:39264->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116139159 (26h37m14.897072524s) [noise=true , address=104.36.82.54:4188 , piece_id=KHF2WI46UOARXIKF4XMR6XOWQDVZNJVB6OCOHKTZVGUQLK4ZOCBA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: write tcp 159.223.143.39:42668->104.36.82.54:4188: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116154418 (26h37m14.897087806s) [upload=25392]: Next segment err: context deadline exceeded Jul 28 00:28:59.116416818 (26h37m14.897350203s) [noise=true , address=104.36.82.11:4146 , piece_id=PSXW7USYOAGXRS6TB5IW3OCU3I5IYCWVFPECOPFOG4QVJ253AYLA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1Sv7eCy62QTiVT1c7mxzJSTAnrCsDoRrxqxfL6govEFRJRQJ6V, address:104.36.82.11:4146): protocol: write tcp 159.223.143.39:46928->104.36.82.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116444787 (26h37m14.897378172s) [noise=true , address=104.36.82.52:4103 , piece_id=HGMKHYXLCGVCMRFMPYGFZRT2FZEG6TBTN7MDI5YRCFH47U6QPLXQ, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:35438->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116515683 (26h37m14.897449048s) [noise=true , address=104.36.82.3:4774 , piece_id=PWWCEVRTZCYVTAEZA6UXC4ZNWCBKEA37H2IYKFNYBOLCRN5VO62A, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1r3Ty7Q8v3b4nT3cVVzQqGDFpHdBu1XKPQvBy221Y78hPzv4Y9, address:104.36.82.3:4774): protocol: write tcp 159.223.143.39:39758->104.36.82.3:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116525789 (26h37m14.897459166s) [noise=true , address=104.36.82.49:4000 , piece_id=UXTZK6FUCF6GL2QYN46YSOMR5TEB2VTK2DF6KEVTJEF2YJHUGK5Q, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12chhfe1weiPC1jjv8vbScP7uFVNXFr6s3UcNRkH8gsSAtGD16D, address:104.36.82.49:4000): protocol: write tcp 159.223.143.39:60962->104.36.82.49:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116592531 (26h37m14.897525898s) [noise=true , address=104.36.82.20:4440 , piece_id=YK6VCCR66N2L37VP2I3AJK3LCCJVQ4IX524RFSHXBWU3S2I2HP3A, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: rpc: dial tcp 104.36.82.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116611347 (26h37m14.897544713s) [noise=true , address=104.36.82.10:4087 , piece_id=W26UBFWTZMOJW4GPIF5LUULHBBI3JWMRFPXUKFDGXSPLXR4VUUYA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:4087): protocol: write tcp 159.223.143.39:53078->104.36.82.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116663839 (26h37m14.897597207s) [noise=true , address=104.36.82.55:4198 , piece_id=USSGLWRZY7MVK45FI54TFU23L4MRMSA5TBCVMCMY6P3AKVX5RGRQ, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:33154->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116715048 (26h37m14.897648416s) [noise=true , address=104.36.82.39:4756 , piece_id=ZWZ4ORCWAW6S63YSYJYA7CAZE7HSXAVHBRG74M336BZFT6PHN2GQ, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12pLAnfGYevL1h7SQBQ6yVM6CexBjC8EvHaj4Lvzj8aXGUfo1Hu, address:104.36.82.39:4756): protocol: rpc: dial tcp 104.36.82.39:4756: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116725581 (26h37m14.897658957s) [noise=true , address=104.36.82.9:4029 , piece_id=KHSTAU73KWGA36762E2YHGFZB7BBFU2QYMPHNGMF5HHS46HWQY3Q, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12snsP1y5fbxY8XrDoXihzQrsBxJ5Y38C2jSGhHtV1ukN426eiP, address:104.36.82.9:4029): protocol: write tcp 159.223.143.39:40562->104.36.82.9:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116781961 (26h37m14.897715325s) [noise=true , address=104.36.82.44:4903 , piece_id=7PDRI2JXUOIIRMKZPNN3FFFYU7CU2O6D2GLTFL3DFXP7GZDY6C5A, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: write tcp 159.223.143.39:43012->104.36.82.44:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116802851 (26h37m14.897736217s) [noise=true , address=104.36.82.41:4775 , piece_id=OLZV77VGFYFHY7N4COUPT4WZ6BO67Q6IGGLND3AHYCTZEU6FCOSA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj, address:104.36.82.41:4775): protocol: rpc: dial tcp 104.36.82.41:4775: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116842231 (26h37m14.897775607s) [noise=true , address=104.36.82.21:4489 , piece_id=KXOJFTS5NBE4MWXWPXY2RUYWUD7UJGMNLPMMQNTAJLUFEUTTY3NA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: write tcp 159.223.143.39:52920->104.36.82.21:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116875619 (26h37m14.897808996s) [noise=true , address=104.36.82.56:4241 , piece_id=S4AWYAUXNB5SIGJYJPBNAWIN77IJHWSNFZCCXPQOBAR7AMMMPYKA, seg_pos={0 0}, upload=25392]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 159.223.143.39:53890->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:59.116905407 (26h37m14.897838782s) [seg_pos={0 0}, upload=25392]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.036062795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X...XX.......X........] duration: 472.040833ms FAIL (2025-07-28 00:28:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.074115ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.233577224s FAIL (2025-07-28 00:28:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.23360919s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X........X.............X] duration: 30.029116796s FAIL (2025-07-28 00:28:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 46996s) [noise=true , address=104.36.82.10:4077 , piece_id=BRLWSXDW3ASYTYMMYCGHUPR662S5FGGV2JJHWZMWZ5XGTBXRPHDA, seg_pos={0 0}, upload=25373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: context canceled; context canceled Jul 28 00:26:41.005236556 (26h34m56.786169939s) [seg_pos={0 0}, upload=25373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:26:41.005340903 (26h34m56.786274281s) [upload=25373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:26:41.005349654 (26h34m56.786283042s) [upload=25373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:26:41.054515340 (26h34m56.835448724s) [upload=25373]: Done uploading segments... Jul 28 00:26:41.054550036 (26h34m56.835483424s) [upload=25373]: Upload finished. err: <nil> Jul 28 00:27:43.642757840 (26h35m59.423691216s) [upload=25387]: Starting upload Jul 28 00:27:43.642880306 (26h35m59.423813668s) [upload=25387]: Done starting upload Jul 28 00:27:43.643072848 (26h35m59.424006218s) [upload=25387]: Uploading segments... Jul 28 00:27:43.643553871 (26h35m59.42448725s) [upload=25387]: Got next segment. Inline: false Jul 28 00:27:43.644426244 (26h35m59.42535963s) [upload=25387]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:27:43.644438712 (26h35m59.425372097s) [upload=25387]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:27:43.674155223 (26h35m59.455088599s) [seg_pos={0 0}, upload=25387]: Begin upload segment... Jul 28 00:27:43.674218316 (26h35m59.45515169s) [seg_pos={0 0}, upload=25387]: Done begin upload segment. Jul 28 00:27:43.674280817 (26h35m59.455214203s) [noise=true , address=45.91.116.14:4191 , piece_id=ZULNV52ZCXOUHYQ6FRYRUMX2CY4GD5ZEYZ6HLQQFDPT2TXHSZG2A, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.674299651 (26h35m59.455233032s) [noise=true , address=45.91.116.11:4118 , piece_id=DMYUFVHB5XO4OBZ7GOAQX2U5AQZ7GBJIPP4KIHWSDDGRI3BBD6VA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.674848339 (26h35m59.45578172s) [noise=true , address=77.90.31.21:4879 , piece_id=GGY2B7FXM3NNWW3ZSHJSEY6NM5KFXV6DLS7AKZOJJWD7CCYBU6LA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.674904311 (26h35m59.455837697s) [noise=true , address=104.36.82.4:4804 , piece_id=GA3Y6N5FEBYOQQZXZ4WJA3DJCXEYLOIGQRUP7XFC25MUIERGEFOA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.675282322 (26h35m59.456215702s) [noise=true , address=77.90.31.13:4626 , piece_id=AUIE5KIAQMNRVZ4G7VMWXJPUP4P2OOOHJPDTVECB2FNR3UQXDDEA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.675393851 (26h35m59.456327229s) [noise=true , address=45.91.116.13:4154 , piece_id=MG4NVO4MDRMOHBOKO6QUNIFRPSZTBN77ZTVJ5XAURHR7DXRUCBEA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.675617458 (26h35m59.456550836s) [noise=true , address=45.91.116.7:4009 , piece_id=UIPSNUA5NZH5NN223MQEZY7KCHWTST5RKIO7FV7SOVLQPPEKNN6Q, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.675991388 (26h35m59.456924769s) [noise=true , address=45.91.116.9:4263 , piece_id=I34TAXNBJK7P4DZDAKVC5YTE36V4RJJPSQCBR3ZNOE76LF5RFFMA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.676530511 (26h35m59.457463889s) [noise=true , address=104.36.82.16:4295 , piece_id=HP6LCPYG7KSE4YG6DWE2CLJD6HQUC6MYORSR2EXIRINIEMS32UXA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.676729685 (26h35m59.457663077s) [noise=true , address=45.91.116.6:4974 , piece_id=VSAXOLXKW4HDE3MTHB6TYS7D5PO47VMPAZ62C4TR5OB77ZT2YB6A, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.677060845 (26h35m59.457994233s) [noise=true , address=45.91.116.8:4045 , piece_id=ZVGC52W5CT5V4TNY7UEKTP5D4OAHURMGD4QIXTIXWI7SFLPSLXJA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.677439810 (26h35m59.458373197s) [noise=true , address=45.91.116.4:4939 , piece_id=Y33KP5446EJUWE3WWZ36QN35VWQVHO5G7KL5ECHK3IFA53AULSFA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.677506604 (26h35m59.458439983s) [noise=true , address=104.36.82.10:4077 , piece_id=YBQTQBZR6SS52QBYBVQTILVWMHKO45PYKW5LJ2VUI24DMX4BZPXA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.677827110 (26h35m59.458760498s) [noise=true , address=104.36.82.8:4972 , piece_id=F4YAV5GF5G3GGPHDGVM47TUYJPOBNRO467BOLB3D3AG37MBKXU2Q, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.678234258 (26h35m59.45916765s) [noise=true , address=45.91.116.12:4226 , piece_id=U4TXFMBUOHEQGKNK5ND3OBQNPKWW2Z3WO5CBSTIRLB3OVJCHYFNA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.678837128 (26h35m59.459770517s) [noise=true , address=45.91.116.8:4047 , piece_id=NMUX52GQZQTRYKHNRT5SKJPF5WUJVGSAEWCEI33IL4NIQ6HBX2AA, seg_pos={0 0}, upload=25387]: Uploading piece... Jul 28 00:27:43.804149945 (26h35m59.585083333s) [upload=25387]: Next returned nil segment Jul 28 00:27:43.804170852 (26h35m59.585104239s) [upload=25387]: Waiting for error group managing segments... Jul 28 00:27:43.809579384 (26h35m59.590512765s) [noise=true , address=45.91.116.7:4009 , piece_id=UIPSNUA5NZH5NN223MQEZY7KCHWTST5RKIO7FV7SOVLQPPEKNN6Q, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.810033543 (26h35m59.590966923s) [noise=true , address=45.91.116.13:4154 , piece_id=MG4NVO4MDRMOHBOKO6QUNIFRPSZTBN77ZTVJ5XAURHR7DXRUCBEA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.810272062 (26h35m59.591205444s) [noise=true , address=45.91.116.14:4191 , piece_id=ZULNV52ZCXOUHYQ6FRYRUMX2CY4GD5ZEYZ6HLQQFDPT2TXHSZG2A, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.810586072 (26h35m59.59151944s) [noise=true , address=45.91.116.6:4974 , piece_id=VSAXOLXKW4HDE3MTHB6TYS7D5PO47VMPAZ62C4TR5OB77ZT2YB6A, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.811106726 (26h35m59.592040115s) [noise=true , address=45.91.116.8:4045 , piece_id=ZVGC52W5CT5V4TNY7UEKTP5D4OAHURMGD4QIXTIXWI7SFLPSLXJA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.811585596 (26h35m59.592518967s) [noise=true , address=45.91.116.8:4047 , piece_id=NMUX52GQZQTRYKHNRT5SKJPF5WUJVGSAEWCEI33IL4NIQ6HBX2AA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.811974366 (26h35m59.592907746s) [noise=true , address=77.90.31.21:4879 , piece_id=GGY2B7FXM3NNWW3ZSHJSEY6NM5KFXV6DLS7AKZOJJWD7CCYBU6LA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.812210614 (26h35m59.593144003s) [noise=true , address=45.91.116.4:4939 , piece_id=Y33KP5446EJUWE3WWZ36QN35VWQVHO5G7KL5ECHK3IFA53AULSFA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.812692074 (26h35m59.593625455s) [noise=true , address=77.90.31.13:4626 , piece_id=AUIE5KIAQMNRVZ4G7VMWXJPUP4P2OOOHJPDTVECB2FNR3UQXDDEA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.813491754 (26h35m59.594425124s) [noise=true , address=45.91.116.12:4226 , piece_id=U4TXFMBUOHEQGKNK5ND3OBQNPKWW2Z3WO5CBSTIRLB3OVJCHYFNA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.814038618 (26h35m59.594971987s) [noise=true , address=45.91.116.9:4263 , piece_id=I34TAXNBJK7P4DZDAKVC5YTE36V4RJJPSQCBR3ZNOE76LF5RFFMA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:27:43.814952501 (26h35m59.595885868s) [noise=true , address=45.91.116.11:4118 , piece_id=DMYUFVHB5XO4OBZ7GOAQX2U5AQZ7GBJIPP4KIHWSDDGRI3BBD6VA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: <nil> Jul 28 00:28:13.606529474 (26h36m29.38746285s) [noise=true , address=104.36.82.8:4972 , piece_id=F4YAV5GF5G3GGPHDGVM47TUYJPOBNRO467BOLB3D3AG37MBKXU2Q, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:28:13.606573791 (26h36m29.387507171s) [noise=true , address=104.36.82.10:4077 , piece_id=YBQTQBZR6SS52QBYBVQTILVWMHKO45PYKW5LJ2VUI24DMX4BZPXA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:42056->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606804935 (26h36m29.387738302s) [noise=true , address=104.36.82.4:4804 , piece_id=GA3Y6N5FEBYOQQZXZ4WJA3DJCXEYLOIGQRUP7XFC25MUIERGEFOA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:39334->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606835386 (26h36m29.387768763s) [noise=true , address=104.36.82.16:4295 , piece_id=HP6LCPYG7KSE4YG6DWE2CLJD6HQUC6MYORSR2EXIRINIEMS32UXA, seg_pos={0 0}, upload=25387]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:35574->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:28:13.606862776 (26h36m29.387796142s) [seg_pos={0 0}, upload=25387]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:28:13.633228465 (26h36m29.414161847s) [upload=25387]: Done uploading segments... Jul 28 00:28:13.633293380 (26h36m29.414226768s) [upload=25387]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.029144687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X...XX.......X.......] duration: 481.685164ms FAIL (2025-07-28 00:28:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 481.722946ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [.....................................XXX.XXXXXXXXXXXXXXXXXXX] duration: 1m0.031945229s FAIL (2025-07-28 00:27:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ue , address=104.36.82.15:4264 , piece_id=AS5I65HFXHH7ZFN62K22SKASKPIDOFZ3UDS6KAKVL4NJ3NQYSC5Q, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.089184067 (26h35m14.870117447s) [noise=true , address=104.36.82.9:4008 , piece_id=KISCQ2OM77KVEUWDBZDQHAPEHATO5TIUBAEEOJXAPAKV7ZPR7WKQ, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.089539083 (26h35m14.870472467s) [noise=true , address=104.36.82.41:4772 , piece_id=FWR4EYEML6IJ3GH3EIIUMZ6U6RCJB5KHISMEF2IDXKXPRJ7FOBOQ, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.089899298 (26h35m14.870832679s) [noise=true , address=104.36.82.21:4493 , piece_id=QFT4BMUL7AGXDSWJDANCLOOV72HEZUJ7D45BXV4JQV42HI2EQYNA, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.090244807 (26h35m14.871178185s) [noise=true , address=104.36.82.44:4904 , piece_id=XBFXHBAKYTIDAKX4UCY2F2WZDDDJSU3SS5BAAGXC2466T4VYHMAQ, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.090558870 (26h35m14.87149225s) [noise=true , address=104.36.82.4:4808 , piece_id=H5PYMC5FQGDCOIPUPU7TH3DROFRUQUXCAF7HWIO7YDGCWFSNEMFA, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.090893976 (26h35m14.871827368s) [noise=true , address=104.36.82.16:4307 , piece_id=JYF5CLE4SUWNHWN35VKL2I2F6GPRRUUZDSK3G5GMKW6J4XQ5GXSQ, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.091345654 (26h35m14.872279029s) [noise=true , address=104.36.82.54:4170 , piece_id=Z5ZPG5VOA6PQK5YOOEBSNEEJV4NGA5CTF6CFLHIXNW3P63MZT4XQ, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.091644517 (26h35m14.872577888s) [noise=true , address=104.36.82.10:4098 , piece_id=KY6YKWWBDAXX6AKPBGJLYUR2MCTU4QH2VMEJFEVWQ5X6IVAH5SWQ, seg_pos={0 0}, upload=25377]: Uploading piece... Jul 28 00:26:59.584442781 (26h35m15.36537616s) [upload=25377]: Next returned nil segment Jul 28 00:26:59.584455323 (26h35m15.365388706s) [upload=25377]: Waiting for error group managing segments... Jul 28 00:26:59.753380672 (26h35m15.534314048s) [noise=true , address=64.49.13.169:4711 , piece_id=6QTAF5H2H3STOVYYD6CTF2QHULPAL5HPAMJW54YY7JMA3OG7XM4Q, seg_pos={0 0}, upload=25377]: Done uploading piece. err: <nil> Jul 28 00:27:00.131959188 (26h35m15.91289257s) [noise=true , address=79.127.205.234:4619 , piece_id=VTDJZ6AQ22CH2BUZ52U5GBIR23O4I5A6KWOAOEYEXOC5L7FXKT4Q, seg_pos={0 0}, upload=25377]: Done uploading piece. err: <nil> Jul 28 00:27:59.018147531 (26h36m14.799080909s) [noise=true , address=104.36.82.54:4170 , piece_id=Z5ZPG5VOA6PQK5YOOEBSNEEJV4NGA5CTF6CFLHIXNW3P63MZT4XQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: rpc: dial tcp 104.36.82.54:4170: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.018448200 (26h36m14.799381578s) [noise=true , address=104.36.82.9:4008 , piece_id=KISCQ2OM77KVEUWDBZDQHAPEHATO5TIUBAEEOJXAPAKV7ZPR7WKQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.018568336 (26h36m14.799501717s) [noise=true , address=104.36.82.21:4493 , piece_id=QFT4BMUL7AGXDSWJDANCLOOV72HEZUJ7D45BXV4JQV42HI2EQYNA, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: rpc: dial tcp 104.36.82.21:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.018651112 (26h36m14.799584492s) [noise=true , address=104.36.82.44:4904 , piece_id=XBFXHBAKYTIDAKX4UCY2F2WZDDDJSU3SS5BAAGXC2466T4VYHMAQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:104.36.82.44:4904): protocol: rpc: dial tcp 104.36.82.44:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.018722526 (26h36m14.799655904s) [noise=true , address=104.36.82.4:4808 , piece_id=H5PYMC5FQGDCOIPUPU7TH3DROFRUQUXCAF7HWIO7YDGCWFSNEMFA, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1fbQCaGY9ZssBzyNusnhyY994e8ytvW4sQDQRt7q2tdGrg6ps3, address:104.36.82.4:4808): protocol: rpc: dial tcp 104.36.82.4:4808: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.018907780 (26h36m14.799841158s) [noise=true , address=104.36.82.46:4493 , piece_id=P77IP5VNBNG2VY57YW2HAXC6WN3JFE6A2KLMH7FIWH2UIGT53FBA, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 159.223.143.39:35816->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019102313 (26h36m14.800035696s) [noise=true , address=104.36.82.56:4246 , piece_id=D2LAVYEITN2C6AMCXTEV3IFKP5XAKJLKTV6FGWL7EA5WP4SUGPFQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1XtKegw9zcPDGZu9z7STKgAzwx5KFrkB9qCgdNUe6XqZ6tzzj9, address:104.36.82.56:4246): protocol: rpc: dial tcp 104.36.82.56:4246: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019186192 (26h36m14.800119563s) [noise=true , address=104.36.82.42:4828 , piece_id=QY77ZZ5IXHJKAIIITBWAFN64SAJ56PWV2ZLQ4QT4UN26HPQPOE5A, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1cX7FtKC8KMopnX9zvkT6fYdnutK84cFjLW6xMwpXuGsqpiN8k, address:104.36.82.42:4828): protocol: write tcp 159.223.143.39:36302->104.36.82.42:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019258791 (26h36m14.800192157s) [noise=true , address=104.36.82.12:4180 , piece_id=EJ2MSXURU3YLPYVG2BQEHF2LJI46Z7FQWRGPMXNLZFX4H7COPJ2Q, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: rpc: dial tcp 104.36.82.12:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019315739 (26h36m14.800249105s) [noise=true , address=104.36.82.7:4915 , piece_id=4CNOIORDFW6FJNIZV7NI2F4H52CEMACZOVVEDZKKNW2WK3EHLYSA, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:104.36.82.7:4915): protocol: rpc: dial tcp 104.36.82.7:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019424748 (26h36m14.800358128s) [noise=true , address=104.36.82.41:4772 , piece_id=FWR4EYEML6IJ3GH3EIIUMZ6U6RCJB5KHISMEF2IDXKXPRJ7FOBOQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1zKCizjMAKNzq6CvZaVBJmjox6abJzywmtbjCUS1dWcDV2qSip, address:104.36.82.41:4772): protocol: rpc: dial tcp 104.36.82.41:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019544771 (26h36m14.800478161s) [noise=true , address=104.36.82.15:4264 , piece_id=AS5I65HFXHH7ZFN62K22SKASKPIDOFZ3UDS6KAKVL4NJ3NQYSC5Q, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1ovYsPwYdx7AYPboFx3Xdcg4VQrPGZj2XTMTqgox6zm81gbf8Z, address:104.36.82.15:4264): protocol: rpc: dial tcp 104.36.82.15:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019736250 (26h36m14.800669638s) [noise=true , address=104.36.82.16:4307 , piece_id=JYF5CLE4SUWNHWN35VKL2I2F6GPRRUUZDSK3G5GMKW6J4XQ5GXSQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:12QGfzeTEErAaE5YRTLi1w7SZoJbo2xVKb1Sykj1pW5fbCZYSCn, address:104.36.82.16:4307): protocol: rpc: dial tcp 104.36.82.16:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019815444 (26h36m14.800748822s) [noise=true , address=104.36.82.10:4098 , piece_id=KY6YKWWBDAXX6AKPBGJLYUR2MCTU4QH2VMEJFEVWQ5X6IVAH5SWQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: rpc: dial tcp 104.36.82.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.019916227 (26h36m14.800849609s) [noise=true , address=104.36.82.11:4139 , piece_id=NR2FCFJPWTV7RQAQWSOITZZJY5322HA2R37KD57F3P2C5LDX7PUQ, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: write tcp 159.223.143.39:46254->104.36.82.11:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:27:59.020084213 (26h36m14.801017594s) [noise=true , address=104.36.82.3:4792 , piece_id=6YCWXTN2PDMAL3G7Q7CHUMCAWH2EOOT4WJSWL5VFZ6UZFPTYP3SA, seg_pos={0 0}, upload=25377]: Done uploading piece. err: ecclient: upload failed (node:1fokaL7frEkFdVZRGmqaq3TmZVNYpt4agWCJZ8ABHfsYwkjAuv, address:104.36.82.3:4792): protocol: rpc: dial tcp 104.36.82.3:4792: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 28 00:27:59.020131218 (26h36m14.801064596s) [seg_pos={0 0}, upload=25377]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:27:59.046970662 (26h36m14.827904041s) [upload=25377]: Done uploading segments... Jul 28 00:27:59.046999253 (26h36m14.827932637s) [upload=25377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.031995386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X...XX.......X......] duration: 600.252469ms FAIL (2025-07-28 00:27:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 600.300959ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 466.61836ms FAIL (2025-07-28 00:27:04 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 466.651814ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [......................................XXX.XXXXXXXXXXXXXXXXXX] duration: 1m0.032962973s FAIL (2025-07-28 00:26:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 4m12.934482379s) [noise=true , address=104.36.82.11:4142 , piece_id=JCSRBI2UAJFS2XK7SZJG7Q67BB6JBQSWIUXEJPBYTYCWP3E347VQ, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.153723615 (26h34m12.934656994s) [noise=true , address=104.36.82.49:4995 , piece_id=KHPWECBHBRXA6UIIPOIK6WKYRH5CAVI3EBIPAEJX3QIJCASL6E5A, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.154121880 (26h34m12.935055264s) [noise=true , address=79.127.205.235:4682 , piece_id=RMLTC5GLBCNJUOC5U7AEC2AJUGKG5Q4DNRELGJHWYWGOV7LJ2IWQ, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.154131038 (26h34m12.935064434s) [noise=true , address=104.36.82.53:4127 , piece_id=FYPRNM7NOMWG3Y4AOJHOTLMX75YWMG34K5JRMARXNKPBJAXK2SJA, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.154488391 (26h34m12.935421775s) [noise=true , address=104.36.82.20:4445 , piece_id=XPRJYXTT2S3WBWIO7WR4JZO53FCW55Q6DADPFXR7JAUBGCRCKLOA, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.154712488 (26h34m12.935645881s) [noise=true , address=104.36.82.10:4084 , piece_id=27R4WQIRN5KJ7XMP3TQJTZG64QVEBPFFRPPBXZB6Z3SWOT5X6AUQ, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.155241634 (26h34m12.936175022s) [noise=true , address=104.36.82.40:4960 , piece_id=VO242J5OAWDYRQIOZVLPB6H3YTD42IRJ3KKCWLC46A5YU4JD5P5Q, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.155707270 (26h34m12.936640664s) [noise=true , address=104.36.82.47:4534 , piece_id=A2YTTDI362OSRYELUEE24577IFKWURBL66OP3FJ4EURACDLGKG7Q, seg_pos={0 0}, upload=25361]: Uploading piece... Jul 28 00:25:57.649156212 (26h34m13.430089597s) [upload=25361]: Next returned nil segment Jul 28 00:25:57.649188105 (26h34m13.430121489s) [upload=25361]: Waiting for error group managing segments... Jul 28 00:25:57.695460364 (26h34m13.476393745s) [noise=true , address=64.49.13.140:4508 , piece_id=27ZOQ64XT32CMI2URI5V2QXB26RGRXAKPPYKDPVHC7WIW5EI3SHA, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 28 00:25:58.681995387 (26h34m14.462928757s) [noise=true , address=79.127.205.235:4682 , piece_id=RMLTC5GLBCNJUOC5U7AEC2AJUGKG5Q4DNRELGJHWYWGOV7LJ2IWQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: <nil> Jul 28 00:26:57.070413746 (26h35m12.851347134s) [noise=true , address=104.36.82.53:4127 , piece_id=FYPRNM7NOMWG3Y4AOJHOTLMX75YWMG34K5JRMARXNKPBJAXK2SJA, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: rpc: dial tcp 104.36.82.53:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.070505149 (26h35m12.851438512s) [noise=true , address=104.36.82.54:4165 , piece_id=ILGXRMHNGTROKRACWUDPAIVZ3XZNKOOSZJZUGI2D23Y7IM4DIWVA, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.070568402 (26h35m12.851501775s) [noise=true , address=104.36.82.51:4055 , piece_id=LDZEVRJCGKRMJDYJFGUSFQG7OGYQ7ZNS4WPLQULHI4CFNXVSXBZQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1SAWKJT2JEb9ReJUGDZ545Rp7LeQoCJKYsQgmrDA9csufPf8us, address:104.36.82.51:4055): protocol: rpc: dial tcp 104.36.82.51:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.070630408 (26h35m12.851563781s) [noise=true , address=104.36.82.14:4229 , piece_id=CCXBEB3JCS6LXLDBXCFZM7A6QYT3INNMZFI6RI5SCBZ3MLXMYH3A, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:12uiCMvJ7pnwXfN57CVu8zR73tikwUZiXr3sG7Xotd9w6KVnViL, address:104.36.82.14:4229): protocol: write tcp 159.223.143.39:34764->104.36.82.14:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.070782254 (26h35m12.851715623s) [noise=true , address=104.36.82.40:4960 , piece_id=VO242J5OAWDYRQIOZVLPB6H3YTD42IRJ3KKCWLC46A5YU4JD5P5Q, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1eH9JpLRcTmvhYuj34kuT8EKeAxBDf7qcKUJ78ZEZPXCAunZL6, address:104.36.82.40:4960): protocol: write tcp 159.223.143.39:52174->104.36.82.40:4960: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.070960930 (26h35m12.851894306s) [noise=true , address=104.36.82.13:4195 , piece_id=VA3IFLODPP3KBVTO7C5XYWXNYWD65QNQTJ3CRE4GY2XLYUW6NKYQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:104.36.82.13:4195): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:26:57.071107293 (26h35m12.85204066s) [noise=true , address=104.36.82.10:4084 , piece_id=27R4WQIRN5KJ7XMP3TQJTZG64QVEBPFFRPPBXZB6Z3SWOT5X6AUQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1mELnhMiRJ5GkUWYEggdxhCQ2JnDbT7qEMSiAuMQsF7BTnwP1u, address:104.36.82.10:4084): protocol: rpc: dial tcp 104.36.82.10:4084: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071186290 (26h35m12.852119654s) [noise=true , address=104.36.82.11:4142 , piece_id=JCSRBI2UAJFS2XK7SZJG7Q67BB6JBQSWIUXEJPBYTYCWP3E347VQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071225421 (26h35m12.852158797s) [noise=true , address=104.36.82.50:4013 , piece_id=Y57OTNWEVTCFKJYQE4H6GTBHINXH5VOYOURR4NRUQO7QLORSMREQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: write tcp 159.223.143.39:58974->104.36.82.50:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071254618 (26h35m12.852187982s) [noise=true , address=104.36.82.20:4445 , piece_id=XPRJYXTT2S3WBWIO7WR4JZO53FCW55Q6DADPFXR7JAUBGCRCKLOA, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1zCSLthypJ1VkjhPxE5bTo35d9kUgQch1Wbj8MU7tK4khzCbcN, address:104.36.82.20:4445): protocol: rpc: dial tcp 104.36.82.20:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071305346 (26h35m12.852238708s) [noise=true , address=104.36.82.4:4819 , piece_id=MFFEHDC6RMWR53VFJPCMG3PFLKNJJFTJBCOAJEWYIZCYW5LIFKNA, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:12mh9u38YmD7NcPA5uB1KLaw8dToTUKywJsi1MhhSc17LXo15Gb, address:104.36.82.4:4819): protocol: rpc: dial tcp 104.36.82.4:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071322905 (26h35m12.85225627s) [noise=true , address=104.36.82.47:4534 , piece_id=A2YTTDI362OSRYELUEE24577IFKWURBL66OP3FJ4EURACDLGKG7Q, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: write tcp 159.223.143.39:47640->104.36.82.47:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071375776 (26h35m12.852309137s) [noise=true , address=104.36.82.49:4995 , piece_id=KHPWECBHBRXA6UIIPOIK6WKYRH5CAVI3EBIPAEJX3QIJCASL6E5A, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1F5oTP56RsDAA7Yde8hYvq7Lx5tiFDUHoBvPrYnhQiFy2Rko3N, address:104.36.82.49:4995): protocol: rpc: dial tcp 104.36.82.49:4995: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071420215 (26h35m12.852353581s) [noise=true , address=104.36.82.12:4159 , piece_id=6QZVC7TTMN3M76IPT32GXZ6YOECETWN5FRLUZLDZGGTLOUZBLXKQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: rpc: dial tcp 104.36.82.12:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071433679 (26h35m12.852367056s) [noise=true , address=104.36.82.41:4793 , piece_id=INUHN74OAHCVS3VVKNHASLAAN2U2VDCE3AKZD6GW56BOVMNLFAUA, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1asDxxBjiU6yNbZ5c6uBmSWQ4QbVs5J3ErLbpKBSwqaptpNhpn, address:104.36.82.41:4793): protocol: rpc: dial tcp 104.36.82.41:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071494852 (26h35m12.852428228s) [noise=true , address=104.36.82.56:4242 , piece_id=F6QRSNGTKGMNXKC7IDYQ6QORW5NKCQFAD4XZYKGI6TGHT4Y63AQQ, seg_pos={0 0}, upload=25361]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: write tcp 159.223.143.39:53066->104.36.82.56:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:26:57.071546613 (26h35m12.852479994s) [seg_pos={0 0}, upload=25361]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:26:57.101087979 (26h35m12.882021355s) [upload=25361]: Done uploading segments... Jul 28 00:26:57.101118039 (26h35m12.882051419s) [upload=25361]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.032989684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X........X.............] duration: 426.239827ms FAIL (2025-07-28 00:26:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 426.303218ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 265.483333ms FAIL (2025-07-28 00:26:16 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 265.511246ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X...XX.......X.....] duration: 643.260078ms FAIL (2025-07-28 00:26:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 643.290609ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 405.291639ms FAIL (2025-07-28 00:25:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 405.329187ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.......................................XXX.XXXXXXXXXXXXXXXXX] duration: 1m0.026215282s FAIL (2025-07-28 00:25:55 UTC) ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== , address=104.36.82.3:4772 , piece_id=V44H4JMRK5OAVO3LZW264X7S6NANM4IWH673TJDHKZ4WWKKWNGCA, seg_pos={0 0}, upload=25344]: Uploading piece... Jul 28 00:24:55.607913228 (26h33m11.388846634s) [noise=true , address=104.36.82.12:4158 , piece_id=FIOSN7OORAGCLY76IYKMXFOTDYCGWA4Y43QGOY3SC27CRTCM6L3A, seg_pos={0 0}, upload=25344]: Uploading piece... Jul 28 00:24:55.608489536 (26h33m11.389422949s) [noise=true , address=104.36.82.56:4250 , piece_id=6E6YELRGGA45QKU3YVEY2K5AWSQ4VTUWAVORYJDMZI2X5AM2XP6A, seg_pos={0 0}, upload=25344]: Uploading piece... Jul 28 00:24:55.609013274 (26h33m11.389946673s) [noise=true , address=104.36.82.51:4080 , piece_id=37ZOGQTZE3SFXNZ5VWRCLHF2VTJMYH4ENU53GFUYLMTDLJNA54EQ, seg_pos={0 0}, upload=25344]: Uploading piece... Jul 28 00:24:55.609547778 (26h33m11.390481185s) [noise=true , address=104.36.82.14:4247 , piece_id=F5IFZTQKLPI6AIAAC4KCGQHX4WKCYIAVAHOP6ZSHESHLHJ6PJISQ, seg_pos={0 0}, upload=25344]: Uploading piece... Jul 28 00:24:55.609630482 (26h33m11.390563871s) [noise=true , address=104.36.82.9:4014 , piece_id=OAP2QDMFV426IRDWBRY6QG4FX6XB4LGMI4TNXNGULJBGRHTCYDXQ, seg_pos={0 0}, upload=25344]: Uploading piece... Jul 28 00:25:55.533717594 (26h34m11.31465096s) [noise=true , address=104.36.82.3:4772 , piece_id=V44H4JMRK5OAVO3LZW264X7S6NANM4IWH673TJDHKZ4WWKKWNGCA, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:104.36.82.3:4772): protocol: rpc: dial tcp 104.36.82.3:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.533808206 (26h34m11.314741573s) [noise=true , address=104.36.82.9:4014 , piece_id=OAP2QDMFV426IRDWBRY6QG4FX6XB4LGMI4TNXNGULJBGRHTCYDXQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12uK9ULUdABDm5XBP4nh1omWhR5qoQerFsCmL4bQ9dmNespXpTy, address:104.36.82.9:4014): protocol: rpc: dial tcp 104.36.82.9:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.533876613 (26h34m11.314809979s) [noise=true , address=104.36.82.14:4247 , piece_id=F5IFZTQKLPI6AIAAC4KCGQHX4WKCYIAVAHOP6ZSHESHLHJ6PJISQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:104.36.82.14:4247): protocol: write tcp 159.223.143.39:45402->104.36.82.14:4247: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.533988539 (26h34m11.314921929s) [noise=true , address=104.36.82.42:4820 , piece_id=OSJOIF4EZFFIKCP47R4HEBZZ4B3YKU4S4ZAPVE4SOJGXQ5WFFUMA, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh, address:104.36.82.42:4820): protocol: write tcp 159.223.143.39:44620->104.36.82.42:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534074332 (26h34m11.315007714s) [noise=true , address=104.36.82.10:4084 , piece_id=FKG6L2ZXQFPNWMHAFRH7Q6GHHH43YUHCGMWQ4YVMHOCDO4H3FDRQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1mELnhMiRJ5GkUWYEggdxhCQ2JnDbT7qEMSiAuMQsF7BTnwP1u, address:104.36.82.10:4084): protocol: write tcp 159.223.143.39:55060->104.36.82.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534137706 (26h34m11.315071085s) [noise=true , address=104.36.82.44:4901 , piece_id=77TBDRJN66IMA54TYTACOEIHZIPNHVQ2L6GVL4ISNBCFPBQ4TZ7Q, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 159.223.143.39:54320->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534205852 (26h34m11.315139229s) [noise=true , address=104.36.82.53:4139 , piece_id=MKEPPV3DI5U2OWI6CMJSJQWMEFS3GSI5GGZ7GHNS4CCP4UEBY2DQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: write tcp 159.223.143.39:48110->104.36.82.53:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534265842 (26h34m11.315199218s) [noise=true , address=104.36.82.46:4495 , piece_id=DSSCSLK3FUEDSXYYU4CCE7ZCY4L5PJZFFJUZDWLC4UDLOHOHW7MQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: write tcp 159.223.143.39:47596->104.36.82.46:4495: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534329078 (26h34m11.315262455s) [noise=true , address=104.36.82.7:4914 , piece_id=FNWRXKOS23GPDWFOXNGLOFIKVEYLBFWPM7LG2FVDJOOZE54RASBQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1MArPyMrdvyw4VFuArmattTtneNTaN5mt82nfGwY2sKHvQnL5D, address:104.36.82.7:4914): protocol: write tcp 159.223.143.39:52388->104.36.82.7:4914: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534373837 (26h34m11.315307214s) [noise=true , address=104.36.82.11:4118 , piece_id=RNZSY6YR4PVOGGQXVQTEVQV7SMVVQXWWBN4S6MRNURQHF3TEJR3Q, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:104.36.82.11:4118): protocol: write tcp 159.223.143.39:47052->104.36.82.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534459751 (26h34m11.31539312s) [noise=true , address=104.36.82.54:4161 , piece_id=KYRFGCQYUGT4EESZYIWTW26FAMGAMRP66IIBW3K43GJQRVFXZ5DA, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:104.36.82.54:4161): protocol: write tcp 159.223.143.39:36352->104.36.82.54:4161: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534525526 (26h34m11.315458896s) [noise=true , address=104.36.82.55:4214 , piece_id=7DK7ITOSDLRLRI67BMUKO3TFJVWXGRBWM2DFWVJAX54GQLVNP57A, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi, address:104.36.82.55:4214): protocol: write tcp 159.223.143.39:59682->104.36.82.55:4214: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534585544 (26h34m11.315518912s) [noise=true , address=104.36.82.46:4515 , piece_id=ZJ6774UNTIRJ23WE5QXQJ5U5U7PPRMETUWYMOJQDC2DWPFVCKFVQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:58976->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534634168 (26h34m11.315567534s) [noise=true , address=104.36.82.52:4094 , piece_id=RRBUW5LOP7IDE2HHNAZY63RFIAJM64VK5S66YLIK55F3YTFEKODQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 159.223.143.39:53858->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534656531 (26h34m11.315589901s) [upload=25344]: Next segment err: context deadline exceeded Jul 28 00:25:55.534734177 (26h34m11.315667558s) [noise=true , address=104.36.82.56:4250 , piece_id=6E6YELRGGA45QKU3YVEY2K5AWSQ4VTUWAVORYJDMZI2X5AM2XP6A, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12oHdA7jYDKSYwWLB975bJ3Fu27xjfZDC9utpnQpszmYjVz2CdD, address:104.36.82.56:4250): protocol: write tcp 159.223.143.39:38050->104.36.82.56:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534748576 (26h34m11.315681938s) [noise=true , address=104.36.82.21:4497 , piece_id=NQSZLMI73KWQTBMWO4SLQS5V2VZU2Y7XAN4SUVZYNBIGETCP72UQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:104.36.82.21:4497): protocol: write tcp 159.223.143.39:58806->104.36.82.21:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534800109 (26h34m11.315733608s) [noise=true , address=104.36.82.51:4080 , piece_id=37ZOGQTZE3SFXNZ5VWRCLHF2VTJMYH4ENU53GFUYLMTDLJNA54EQ, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: write tcp 159.223.143.39:39702->104.36.82.51:4080: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534838437 (26h34m11.3157718s) [noise=true , address=104.36.82.12:4158 , piece_id=FIOSN7OORAGCLY76IYKMXFOTDYCGWA4Y43QGOY3SC27CRTCM6L3A, seg_pos={0 0}, upload=25344]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: write tcp 159.223.143.39:45458->104.36.82.12:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:25:55.534867430 (26h34m11.315800823s) [seg_pos={0 0}, upload=25344]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.026238806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X........X............] duration: 558.877238ms FAIL (2025-07-28 00:25:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 558.910579ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 262.945208ms FAIL (2025-07-28 00:25:39 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 262.983936ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration: 505.761759ms FAIL (2025-07-28 00:25:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.829511ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 225.485625ms FAIL (2025-07-28 00:25:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 225.520555ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X...XX.......X....] duration: 463.824854ms FAIL (2025-07-28 00:25:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.863375ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X........X..........] duration: 474.584631ms FAIL (2025-07-28 00:24:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.619747ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [........................................XXX.XXXXXXXXXXXXXXXX] duration: 1m0.02894642s FAIL (2025-07-28 00:24:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 1.721040934 (26h31m57.501974316s) [noise=true , address=104.36.82.42:4812 , piece_id=BJRA6O75LCRWGVO6GJUZSBY6NPCMIS7WRAUGGCLBVTELC7YVC4JA, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.721568560 (26h31m57.502501936s) [noise=true , address=104.36.82.46:4499 , piece_id=DBR7WTJKT7YORNV7ZLB573EDB5I7YELSWDWKJUXMWFAKAWLCVNHQ, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.722043121 (26h31m57.502976506s) [noise=true , address=104.36.82.21:4496 , piece_id=DYK3GKNCC3ONKW2DR7MHUAZAHPLR2ZNRK27ZTVHJTR4FKBWMXGAQ, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.723423714 (26h31m57.504357091s) [noise=true , address=104.36.82.12:4154 , piece_id=5LJZQG24F3T7SVZXIE5WHFFJ5D7JT6KRSTMC6C7LV5VN3ZW6HIJA, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.723905535 (26h31m57.504838901s) [noise=true , address=104.36.82.6:4905 , piece_id=33Q36ES42MSC6GGQ33UKPRFWMVSXBVSFMELYPUAD2EA64VO2A7FA, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.724440355 (26h31m57.50537372s) [noise=true , address=104.36.82.55:4215 , piece_id=S3CX37PKTSNLA23PQCHEZXGCUNW24EJIM2HPHOT2JT5XU4DR65SA, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.724950773 (26h31m57.505884157s) [noise=true , address=104.36.82.7:4923 , piece_id=7JKY2FSKB655YTSRYJFJ2GNPY3F6YUHVPS4AAFF2RZD7VA3CLXKA, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:41.725524955 (26h31m57.506458331s) [noise=true , address=193.149.164.143:4694 , piece_id=K5FKF3GXX6W2ODIH6PT6RBQTEKFTKXUTB22IDWBEKI5SHZC447FQ, seg_pos={0 0}, upload=25326]: Uploading piece... Jul 28 00:23:42.290243096 (26h31m58.071176463s) [upload=25326]: Next returned nil segment Jul 28 00:23:42.290260102 (26h31m58.07119347s) [upload=25326]: Waiting for error group managing segments... Jul 28 00:23:42.650347959 (26h31m58.431281335s) [noise=true , address=193.149.164.143:4694 , piece_id=K5FKF3GXX6W2ODIH6PT6RBQTEKFTKXUTB22IDWBEKI5SHZC447FQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: <nil> Jul 28 00:24:41.638917621 (26h32m57.419850998s) [noise=true , address=104.36.82.11:4126 , piece_id=O2VSWHQ3O75XLGNRAUVDIVK3C3L6O66W2GM4MLESXDZ6CZCDREZQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: write tcp 159.223.143.39:34326->104.36.82.11:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.639733288 (26h32m57.420666654s) [noise=true , address=104.36.82.4:4831 , piece_id=MSMTLN5DOWM6PIDSF44DVVZ6EXPZWO6VU34BBK7L6GNGQSPGW4AQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.639813052 (26h32m57.420746419s) [noise=true , address=104.36.82.9:4014 , piece_id=LMGSZPGCNIF3U6D3LQNOKQYMZIWPYJD52EIGHHPEZHHXW4LX2FXQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12uK9ULUdABDm5XBP4nh1omWhR5qoQerFsCmL4bQ9dmNespXpTy, address:104.36.82.9:4014): protocol: rpc: dial tcp 104.36.82.9:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.639881445 (26h32m57.420814813s) [noise=true , address=104.36.82.54:4187 , piece_id=TFDSDP7ZKDKHWRUJYLWLJWCP6HGJVVS375IEU4KQL3K7I4WNLMUQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: rpc: dial tcp 104.36.82.54:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.639932976 (26h32m57.42086634s) [noise=true , address=104.36.82.21:4496 , piece_id=DYK3GKNCC3ONKW2DR7MHUAZAHPLR2ZNRK27ZTVHJTR4FKBWMXGAQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:104.36.82.21:4496): protocol: rpc: dial tcp 104.36.82.21:4496: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.639991771 (26h32m57.420925138s) [noise=true , address=104.36.82.55:4215 , piece_id=S3CX37PKTSNLA23PQCHEZXGCUNW24EJIM2HPHOT2JT5XU4DR65SA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640063603 (26h32m57.420996969s) [noise=true , address=104.36.82.7:4923 , piece_id=7JKY2FSKB655YTSRYJFJ2GNPY3F6YUHVPS4AAFF2RZD7VA3CLXKA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:104.36.82.7:4923): protocol: rpc: dial tcp 104.36.82.7:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640119648 (26h32m57.421053013s) [noise=true , address=104.36.82.49:4011 , piece_id=BAEHSPWXYRA2N4S3PL5UYONUJORE6DPBMOW2JITR46PNDHGBMQZA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: rpc: dial tcp 104.36.82.49:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640484426 (26h32m57.421417804s) [noise=true , address=104.36.82.10:4110 , piece_id=RZJQ5VNVNVRSP4JTZNDIRHRWFWS3DFCT6XK5ZL2GY5MFACBTZZCQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi, address:104.36.82.10:4110): protocol: rpc: dial tcp 104.36.82.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640586650 (26h32m57.421520027s) [noise=true , address=104.36.82.14:4255 , piece_id=63NBVAXFHD3WH35Q7D3ERIECIDPEUKUCQCT7U6P7AD2U6WXZFUTQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: rpc: dial tcp 104.36.82.14:4255: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640606481 (26h32m57.421539848s) [noise=true , address=104.36.82.12:4154 , piece_id=5LJZQG24F3T7SVZXIE5WHFFJ5D7JT6KRSTMC6C7LV5VN3ZW6HIJA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640664319 (26h32m57.421597695s) [noise=true , address=104.36.82.52:4100 , piece_id=S764KNGH6NZGNV5EPFFKLK64PPRUCU5APLHAQSMGNF2635VXS3AA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640674024 (26h32m57.42160739s) [noise=true , address=104.36.82.6:4905 , piece_id=33Q36ES42MSC6GGQ33UKPRFWMVSXBVSFMELYPUAD2EA64VO2A7FA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1qAL8paQ8Z3A9VgPRQ2UUG2LytmYZKdz9hjLzK8w8fQmHiYf88, address:104.36.82.6:4905): protocol: rpc: dial tcp 104.36.82.6:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640750544 (26h32m57.421683923s) [noise=true , address=104.36.82.41:4763 , piece_id=F4MSSB6JGPH4RQOEQECPMIG4WJK7TGWOX4Z6S23OXVN5XIIAU6RQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:104.36.82.41:4763): protocol: write tcp 159.223.143.39:48398->104.36.82.41:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640773894 (26h32m57.421707261s) [noise=true , address=104.36.82.46:4499 , piece_id=DBR7WTJKT7YORNV7ZLB573EDB5I7YELSWDWKJUXMWFAKAWLCVNHQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:24:41.640825641 (26h32m57.421759022s) [noise=true , address=104.36.82.46:4481 , piece_id=XU23ZIORQO7OYELB2AUHIMWEGHKNCKXP5UYGRG7MZFGV2QONLCZQ, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640838746 (26h32m57.421772113s) [noise=true , address=104.36.82.42:4812 , piece_id=BJRA6O75LCRWGVO6GJUZSBY6NPCMIS7WRAUGGCLBVTELC7YVC4JA, seg_pos={0 0}, upload=25326]: Done uploading piece. err: ecclient: upload failed (node:1TmXeVTeejZvrWi2RPxMutM6kXuZiMCtPxQBwL4tq6mhoLwFQo, address:104.36.82.42:4812): protocol: write tcp 159.223.143.39:43516->104.36.82.42:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:24:41.640863038 (26h32m57.421796423s) [seg_pos={0 0}, upload=25326]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:24:41.665990588 (26h32m57.446923965s) [upload=25326]: Done uploading segments... Jul 28 00:24:41.666047138 (26h32m57.446980516s) [upload=25326]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.028966684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 221.638038ms FAIL (2025-07-28 00:24:27 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 221.671761ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X...XX.......X...] duration: 434.175159ms FAIL (2025-07-28 00:24:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.211935ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 236.326289ms FAIL (2025-07-28 00:24:03 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 236.397647ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration: 468.221468ms FAIL (2025-07-28 00:23:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 468.255332ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.........................................XXX.XXXXXXXXXXXXXXX] duration: 1m0.062333332s FAIL (2025-07-28 00:23:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== : Uploading piece... Jul 28 00:22:41.567962568 (26h30m57.348895954s) [noise=true , address=104.36.82.3:4770 , piece_id=REZANEVMXYIR5HQCH4ZAQHDCLRX577PDQHSAT4XGCV3LGBYGCX4A, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.569497158 (26h30m57.350430544s) [noise=true , address=104.36.82.52:4119 , piece_id=YBWGOCDLVRWX5ATL5SA5COXEIRZ5Z4NX7GDZVRSRK2DJCBPVNZJA, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.570220621 (26h30m57.351154003s) [noise=true , address=104.36.82.53:4134 , piece_id=JS2FO7B5OWRWNVZVJXEEXGCERNJOB5QOEUGUXVU7EZ4VYMHXV3VQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.571346077 (26h30m57.352279464s) [noise=true , address=104.36.82.46:4502 , piece_id=36QF4EQS62L34VRP6X4TUKL7PCDZZF3JA3B6LMPNASHRYHCNTFHQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.571810505 (26h30m57.352743885s) [noise=true , address=104.36.82.55:4201 , piece_id=FJLWAFMFMTLOAJ5AYJZD5WRZ5XC3WO6PXL6ZQ64TGWTDJK2X6FPA, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.572394285 (26h30m57.353327667s) [noise=true , address=104.36.82.20:4459 , piece_id=W3BOCNUVVXZP2YXNG354PZSCK4ZOUCFFJAGZPUJSKDDQZGZALQ2Q, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.572830490 (26h30m57.353763873s) [noise=true , address=104.36.82.47:4543 , piece_id=QWT7IB6ALL3EOQNP4XMFJVORLNWNR7SRFKKEPNDGHIR2UEU6ESKQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.573268094 (26h30m57.354201482s) [noise=true , address=104.36.82.14:4228 , piece_id=HM55DPNXQ7K6HCP7PGMATEDGZCNOQW26R6K5VMX426YF3OEY7ISQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.573683414 (26h30m57.354616799s) [noise=true , address=104.36.82.12:4180 , piece_id=OIZPDYBZ6Z46CMNNLSCWBA4DBZIECTGXHM7ICXYXSGPN3V365SQA, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.773343829 (26h30m57.554277212s) [upload=25309]: Next returned nil segment Jul 28 00:22:41.773367760 (26h30m57.554301145s) [upload=25309]: Waiting for error group managing segments... Jul 28 00:22:46.743972499 (26h31m2.524905897s) [noise=true , address=77.90.31.16:4744 , piece_id=IZRQI67AXTSVX6FUORLWEWGSHAGCTQL6XS6Y46F7XYT7YGREWL6Q, seg_pos={0 0}, upload=25309]: Done uploading piece. err: <nil> Jul 28 00:22:47.664618143 (26h31m3.445551518s) [noise=true , address=79.127.201.215:4360 , piece_id=SNER7ANSCR5WPOSBM3FC5J7VOFR3BI3JFJONQLF3X6TMFZXW2FZA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: <nil> Jul 28 00:23:41.496665723 (26h31m57.277599103s) [noise=true , address=104.36.82.47:4547 , piece_id=ST32IUP7KZYZ5JXB6XFHRJM6QWLIZJIKE7Q24IRE5AJOFBQTLTPA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496761937 (26h31m57.277695313s) [noise=true , address=104.36.82.15:4288 , piece_id=L5U6GCRT3CCHLYWWHY3NJOYHGJC2ESCRHWYLUTP56IWXRCYXE5AQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:104.36.82.15:4288): protocol: rpc: dial tcp 104.36.82.15:4288: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496858906 (26h31m57.277792282s) [noise=true , address=104.36.82.12:4180 , piece_id=OIZPDYBZ6Z46CMNNLSCWBA4DBZIECTGXHM7ICXYXSGPN3V365SQA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: rpc: dial tcp 104.36.82.12:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496931683 (26h31m57.277865059s) [noise=true , address=104.36.82.46:4502 , piece_id=36QF4EQS62L34VRP6X4TUKL7PCDZZF3JA3B6LMPNASHRYHCNTFHQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496993364 (26h31m57.277926739s) [noise=true , address=104.36.82.13:4202 , piece_id=KBHUZJLB5KHRRF6RKHLO4S6MIJ3YRFZV7FINM2VITYXVQBNQATJA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:18wf5KnaR6WaR7MJivJkMJ7FejbuR6KnyvtRQgazNKfDb8fbsH, address:104.36.82.13:4202): protocol: rpc: dial tcp 104.36.82.13:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497028360 (26h31m57.277961786s) [noise=true , address=104.36.82.45:4929 , piece_id=VCWYU3LV2FXQN74OGTVXP2WR34FTFTF5ZY22P5FPZSUSWHQGVPKQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm, address:104.36.82.45:4929): protocol: rpc: dial tcp 104.36.82.45:4929: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497075291 (26h31m57.278008666s) [noise=true , address=104.36.82.44:4904 , piece_id=TG4AI2RT63XP6R65YVG6IDCO63TD2CUZXNWNM2EVOVC2FVSXB72A, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:104.36.82.44:4904): protocol: rpc: dial tcp 104.36.82.44:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497146365 (26h31m57.278079753s) [noise=true , address=104.36.82.3:4770 , piece_id=REZANEVMXYIR5HQCH4ZAQHDCLRX577PDQHSAT4XGCV3LGBYGCX4A, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: rpc: dial tcp 104.36.82.3:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497219658 (26h31m57.278153034s) [noise=true , address=104.36.82.21:4494 , piece_id=VCC54HWJ2OZES47SN7KCS7MRAWUY4QZPU2MSAJUMHJ4YL522ZH6Q, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1sRaMuQG18B5SfPpzxp5VWjrwe38SS38g4askmbURxU9tepJHi, address:104.36.82.21:4494): protocol: rpc: dial tcp 104.36.82.21:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497299255 (26h31m57.278232643s) [noise=true , address=104.36.82.53:4134 , piece_id=JS2FO7B5OWRWNVZVJXEEXGCERNJOB5QOEUGUXVU7EZ4VYMHXV3VQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497369485 (26h31m57.278302858s) [noise=true , address=104.36.82.20:4459 , piece_id=W3BOCNUVVXZP2YXNG354PZSCK4ZOUCFFJAGZPUJSKDDQZGZALQ2Q, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: rpc: dial tcp 104.36.82.20:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497397688 (26h31m57.278331067s) [noise=true , address=104.36.82.55:4201 , piece_id=FJLWAFMFMTLOAJ5AYJZD5WRZ5XC3WO6PXL6ZQ64TGWTDJK2X6FPA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497428141 (26h31m57.278361528s) [noise=true , address=104.36.82.47:4543 , piece_id=QWT7IB6ALL3EOQNP4XMFJVORLNWNR7SRFKKEPNDGHIR2UEU6ESKQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: rpc: dial tcp 104.36.82.47:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497441107 (26h31m57.278374471s) [noise=true , address=104.36.82.52:4119 , piece_id=YBWGOCDLVRWX5ATL5SA5COXEIRZ5Z4NX7GDZVRSRK2DJCBPVNZJA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: rpc: dial tcp 104.36.82.52:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497454182 (26h31m57.278387545s) [noise=true , address=104.36.82.15:4274 , piece_id=NP6WPSZVJRKRP2BWZOBD5KXK6QZIBS4A2GBRZIQ3U6AGORQE2UBQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12nZ58ntQujpYk3UVGt4rD6xssGW3RxmvgM84a4Ka8BCRowuaim, address:104.36.82.15:4274): protocol: rpc: dial tcp 104.36.82.15:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497462919 (26h31m57.278396281s) [noise=true , address=104.36.82.14:4228 , piece_id=HM55DPNXQ7K6HCP7PGMATEDGZCNOQW26R6K5VMX426YF3OEY7ISQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 159.223.143.39:37846->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497507499 (26h31m57.278440884s) [seg_pos={0 0}, upload=25309]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:23:41.556138464 (26h31m57.337071835s) [upload=25309]: Done uploading segments... Jul 28 00:23:41.556163895 (26h31m57.337097264s) [upload=25309]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.062370803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X...XX.......X..] duration: 529.188691ms FAIL (2025-07-28 00:23:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.216001ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X...XX.......X.] duration: 470.021029ms FAIL (2025-07-28 00:23:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.049655ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X........X........] duration: 444.152436ms FAIL (2025-07-28 00:22:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.196671ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X...XX.......X] duration: 30.026525096s FAIL (2025-07-28 00:22:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026546783s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 348.546741ms FAIL (2025-07-28 00:22:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 348.58552ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..........................................XXX.XXXXXXXXXXXXXX] duration: 1m0.040319572s FAIL (2025-07-28 00:22:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 25290]: Uploading piece... Jul 28 00:21:41.483328084 (26h29m57.264261467s) [noise=true , address=104.36.82.46:4515 , piece_id=H33YBP3JDSIKRD24YHTS6U6VN6NE5ZXLD5TJBWP3QJTMGDE2UF3A, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.485565755 (26h29m57.266499142s) [noise=true , address=104.36.82.55:4198 , piece_id=33EFCGEMN3PZ2U3PU6QNI6DLM7GAO3VVCMU4T6RZSYXRP4E3XE3Q, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.487630081 (26h29m57.268563457s) [noise=true , address=104.36.82.53:4148 , piece_id=4WZJKIOM3CQARHAC5J54FEM4UXQ6OI72PZI3PWCWWHBDIBHUR3CQ, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.488025843 (26h29m57.268959225s) [noise=true , address=104.36.82.10:4089 , piece_id=7DZ3ZR2WR3NQBGRDUPIPRIPR4ZOCIMLUK25GP3V2LGNRWOXKIRLQ, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.489924722 (26h29m57.27085811s) [noise=true , address=104.36.82.21:4478 , piece_id=DC5YA6BSCF772G4V55TAY2T3EPVGHLIT4OZOHSSRROYQDKNZVB2Q, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.490541848 (26h29m57.271475234s) [noise=true , address=104.36.82.4:4836 , piece_id=SJOG7BPO4O3GWUHBCPKGUZER7J2J6KL67QVFGEMHGPSG5Q66RFDA, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.491926865 (26h29m57.272860253s) [noise=true , address=79.127.201.215:4353 , piece_id=ZWEG2SOV23FK5KS4NNIV3WEQVVTYFMAOAKCJIRE5OFEWGWVBQJQQ, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.492590382 (26h29m57.273523764s) [noise=true , address=79.127.205.234:4619 , piece_id=ONQ6DOMMYONHC7K33WL2HAM6PEJKIGPMJKR7WRPY22XOFLHXR3FA, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.494297212 (26h29m57.275230592s) [noise=true , address=104.36.82.13:4191 , piece_id=T5V4GNFULI4GHYK5BS7KSENNICT2DJI5DNFPVA36IETYSF45Z3DA, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.686457009 (26h29m57.467390387s) [upload=25290]: Next returned nil segment Jul 28 00:21:41.686481493 (26h29m57.467414872s) [upload=25290]: Waiting for error group managing segments... Jul 28 00:21:42.210589749 (26h29m57.991523119s) [noise=true , address=79.127.205.234:4619 , piece_id=ONQ6DOMMYONHC7K33WL2HAM6PEJKIGPMJKR7WRPY22XOFLHXR3FA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: <nil> Jul 28 00:21:43.685485742 (26h29m59.466419107s) [noise=true , address=79.127.201.215:4353 , piece_id=ZWEG2SOV23FK5KS4NNIV3WEQVVTYFMAOAKCJIRE5OFEWGWVBQJQQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: <nil> Jul 28 00:22:41.391099061 (26h30m57.172032437s) [noise=true , address=104.36.82.55:4198 , piece_id=33EFCGEMN3PZ2U3PU6QNI6DLM7GAO3VVCMU4T6RZSYXRP4E3XE3Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: rpc: dial tcp 104.36.82.55:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391487218 (26h30m57.17242062s) [noise=true , address=104.36.82.46:4504 , piece_id=RSC54PFOYEFGK6PSHHI7K746VUE3FS5TO2CC7IAXQQJ7XARQX4HA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391513090 (26h30m57.172446466s) [noise=true , address=104.36.82.11:4132 , piece_id=RGBNEXDA4FEDPNFQZIKJU6NOU27W7XNQ2HNKVR254VGG6EKW2CRA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391596103 (26h30m57.172529477s) [noise=true , address=104.36.82.53:4148 , piece_id=4WZJKIOM3CQARHAC5J54FEM4UXQ6OI72PZI3PWCWWHBDIBHUR3CQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391668207 (26h30m57.172601583s) [noise=true , address=104.36.82.21:4478 , piece_id=DC5YA6BSCF772G4V55TAY2T3EPVGHLIT4OZOHSSRROYQDKNZVB2Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:17kFjjJyvHApPUxUHJ6HeMEBCoo6xTbr8QizXjBEHwWj3mZUha, address:104.36.82.21:4478): protocol: rpc: dial tcp 104.36.82.21:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391676613 (26h30m57.172609991s) [noise=true , address=104.36.82.46:4515 , piece_id=H33YBP3JDSIKRD24YHTS6U6VN6NE5ZXLD5TJBWP3QJTMGDE2UF3A, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: rpc: dial tcp 104.36.82.46:4515: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391737703 (26h30m57.172671077s) [noise=true , address=104.36.82.12:4160 , piece_id=YV425YTIHBFDSGBM4TCDLIFPUXE2MT3EWQM3E4TGWPD4APSGIE6Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12dggZZRv5coYBNJEBvHNqWtMSqYkZoXYmitc9nBKJZftnsX17D, address:104.36.82.12:4160): protocol: rpc: dial tcp 104.36.82.12:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391768098 (26h30m57.172701475s) [noise=true , address=104.36.82.42:4831 , piece_id=6X2DZFR2FJCUAS2EEMKJML3PLPI3YT43JQP6UOIWXFSILG4FWFTA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: rpc: dial tcp 104.36.82.42:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391838699 (26h30m57.172772077s) [noise=true , address=104.36.82.47:4535 , piece_id=Y3PQAYUI2W4CRM3YV2OV4LCRTJYEO4EKGZTZUJ36IRE524UNC22Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391856126 (26h30m57.172789507s) [noise=true , address=104.36.82.2:4742 , piece_id=S3XSIA4P3YH2GQLGXBVJOAYWSYLVYM44CIAO6WV347GMKREXU67Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT, address:104.36.82.2:4742): protocol: rpc: dial tcp 104.36.82.2:4742: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391957135 (26h30m57.17289051s) [noise=true , address=104.36.82.13:4191 , piece_id=T5V4GNFULI4GHYK5BS7KSENNICT2DJI5DNFPVA36IETYSF45Z3DA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:104.36.82.13:4191): protocol: rpc: dial tcp 104.36.82.13:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392034885 (26h30m57.172968399s) [noise=true , address=104.36.82.54:4173 , piece_id=PB7GRW56RRIMQGE3K75I2QJ7DGWVK34ZQKSDPQLAPGMLDN23SNGA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392044423 (26h30m57.172977792s) [noise=true , address=104.36.82.14:4234 , piece_id=ZX6SN77U4F5E5UN6Q7CRCU7PJVWS47H7HZIBDOII7OPSTC4U6AJQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12DtCsEnAhXZnGRXjnhNfqNz4RvPnvTMDz1ur7DyZiQtQ7nbwDS, address:104.36.82.14:4234): protocol: rpc: dial tcp 104.36.82.14:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392056095 (26h30m57.172989472s) [noise=true , address=104.36.82.40:4950 , piece_id=YA2GKHJ5RHSXJ7KGAGRLKKYW5OXT44IBQJ6FOKBLKGTCPGVS2JHQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: rpc: dial tcp 104.36.82.40:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392099925 (26h30m57.173033301s) [noise=true , address=104.36.82.10:4089 , piece_id=7DZ3ZR2WR3NQBGRDUPIPRIPR4ZOCIMLUK25GP3V2LGNRWOXKIRLQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: rpc: dial tcp 104.36.82.10:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392130596 (26h30m57.173063976s) [noise=true , address=104.36.82.4:4836 , piece_id=SJOG7BPO4O3GWUHBCPKGUZER7J2J6KL67QVFGEMHGPSG5Q66RFDA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:104.36.82.4:4836): protocol: write tcp 159.223.143.39:44864->104.36.82.4:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392158883 (26h30m57.173092268s) [seg_pos={0 0}, upload=25290]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:22:41.427412168 (26h30m57.20834553s) [upload=25290]: Done uploading segments... Jul 28 00:22:41.427443367 (26h30m57.20837673s) [upload=25290]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.040342103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration: 450.728661ms FAIL (2025-07-28 00:22:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.755986ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X........X......] duration: 572.916132ms FAIL (2025-07-28 00:22:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 572.951057ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 229.722868ms FAIL (2025-07-28 00:21:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 229.763432ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [...........................................XXX.XXXXXXXXXXXXX] duration: 1m0.025682029s FAIL (2025-07-28 00:21:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.371192168 (26h28m57.152125549s) [noise=true , address=104.36.82.21:4503 , piece_id=BCZTHVIZMU6PPGUU75TYN2XC6VB2JQYBIMMFZLKBU6A73OAUABYA, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.371525656 (26h28m57.152459038s) [noise=true , address=104.36.82.53:4154 , piece_id=MOLL2F7FKVDS2FZDA2CC7KVHOFSIHG5EBPEM6VU5ZHOSWQTYZQRA, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.371849912 (26h28m57.152783291s) [noise=true , address=104.36.82.55:4196 , piece_id=YBSM5ABU4IUSUZ44I2VHWTPVZY6OGBYEZA6FZT2MX6KQHHGARWPQ, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.372252300 (26h28m57.153185691s) [noise=true , address=104.36.82.15:4279 , piece_id=EL7MFXVWOG2SRXX4RA3EXKYJB52TKLUWDHXYEMIJRIC5T525DCMQ, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.372669809 (26h28m57.153603191s) [noise=true , address=104.36.82.2:4755 , piece_id=ZIWQFL2CLETGAQXC4UOPOE7T3Y632KF5RZB5ZUBRX3M6XVLOJF5Q, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.854332036 (26h28m57.635265398s) [upload=25273]: Next returned nil segment Jul 28 00:20:41.854358717 (26h28m57.635292095s) [upload=25273]: Waiting for error group managing segments... Jul 28 00:20:44.216778103 (26h28m59.997711485s) [noise=true , address=77.90.31.16:4748 , piece_id=X336XIAHIYHCFUY2HHBUZ356W4TAV5XEJFZ3OMNXFDA42QH2A6YA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: <nil> Jul 28 00:21:41.293343579 (26h29m57.074276965s) [noise=true , address=104.36.82.15:4279 , piece_id=EL7MFXVWOG2SRXX4RA3EXKYJB52TKLUWDHXYEMIJRIC5T525DCMQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293421082 (26h29m57.074354452s) [noise=true , address=104.36.82.54:4187 , piece_id=7OJZPBJDU2JSHLUA674SBJP3VLE2EHZ3ZZY3XKEY3QFVJ7J6U55Q, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: write tcp 159.223.143.39:47344->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293614435 (26h29m57.07454781s) [noise=true , address=104.36.82.11:4131 , piece_id=TJZQJHUZD4KDRUJ7BGFHJLA7WQAURQNUUNJQAEWEEVAJHFV6PWAA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 159.223.143.39:47068->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293715157 (26h29m57.074648533s) [noise=true , address=104.36.82.21:4503 , piece_id=BCZTHVIZMU6PPGUU75TYN2XC6VB2JQYBIMMFZLKBU6A73OAUABYA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: write tcp 159.223.143.39:41764->104.36.82.21:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293773285 (26h29m57.074706674s) [noise=true , address=104.36.82.53:4154 , piece_id=MOLL2F7FKVDS2FZDA2CC7KVHOFSIHG5EBPEM6VU5ZHOSWQTYZQRA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: write tcp 159.223.143.39:56510->104.36.82.53:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293838795 (26h29m57.07477217s) [noise=true , address=104.36.82.9:4022 , piece_id=MVCAEDRHX5NDCOOQ3KSV3YUTA6X7P3TGRQBGQF6JXRBIIIDEQMJA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: write tcp 159.223.143.39:50654->104.36.82.9:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293938568 (26h29m57.074871943s) [noise=true , address=104.36.82.55:4196 , piece_id=YBSM5ABU4IUSUZ44I2VHWTPVZY6OGBYEZA6FZT2MX6KQHHGARWPQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 159.223.143.39:34154->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293997001 (26h29m57.074930379s) [noise=true , address=104.36.82.2:4755 , piece_id=ZIWQFL2CLETGAQXC4UOPOE7T3Y632KF5RZB5ZUBRX3M6XVLOJF5Q, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 159.223.143.39:43770->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294068450 (26h29m57.075001825s) [noise=true , address=104.36.82.55:4227 , piece_id=PKEMZXJ25F4HSX7B6X4JTYLEIQRWIALETKKLSBM3GUKFVHR47AZA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: write tcp 159.223.143.39:36152->104.36.82.55:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294296983 (26h29m57.075230362s) [noise=true , address=104.36.82.47:4546 , piece_id=7JKKJPM3EJORWD5JWRJXLWU2XBOVORETDQ5B4NPKFAOCOAGF6ANA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: write tcp 159.223.143.39:56188->104.36.82.47:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294391262 (26h29m57.075324637s) [noise=true , address=104.36.82.13:4208 , piece_id=FIKJUW647V3WH762O22OUEHLCYFBUUPMPMVFI5P62IPSKZKAVEWA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12SXSN3HpeYwgHRpFFST5wwP3hhXEe3QB8VzWTNNE19sFoWW4o5, address:104.36.82.13:4208): protocol: write tcp 159.223.143.39:33180->104.36.82.13:4208: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294531549 (26h29m57.075464924s) [noise=true , address=104.36.82.46:4515 , piece_id=NP2EU4N3DIUMZOIWPCYCCBQZ5GHVQYKKSU4Z72GREJALGHZIGZCA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:52480->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294588877 (26h29m57.075522255s) [noise=true , address=104.36.82.20:4444 , piece_id=C3JVJ35W6BAZ7CRRVUJZCJWWHXG3NSWQPHNKGRTO73QR755DNSEQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:4444): protocol: write tcp 159.223.143.39:44272->104.36.82.20:4444: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294635310 (26h29m57.075568687s) [noise=true , address=104.36.82.6:4906 , piece_id=QWM3NL6D3LIDGUII3QRWIOP5W7MGVRDQMRU7DWEGBSB2V4W22QBQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1gByS6jTUkLuHrY8MafgmkmB4wXyY2j9Mt48fkpPNWbCKTcnty, address:104.36.82.6:4906): protocol: write tcp 159.223.143.39:55650->104.36.82.6:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294710608 (26h29m57.07564399s) [noise=true , address=104.36.82.14:4227 , piece_id=JGP5VDKQ7DWJXO2D452CP5GAZZOERKP5TJHG7KQI5X6ND4VQOSOQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:104.36.82.14:4227): protocol: write tcp 159.223.143.39:38254->104.36.82.14:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294771859 (26h29m57.075705238s) [noise=true , address=104.36.82.4:4810 , piece_id=PZ7UD5VUMFVVKWSCDEQ7GLPUT6PWFVIKGLB4UBZCLVQEEXAQGOQQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12GTWz2fWHLAApibeARUerDvJ9c4AWYRVAvumRxgJwwqCYYJJfg, address:104.36.82.4:4810): protocol: write tcp 159.223.143.39:43612->104.36.82.4:4810: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294798168 (26h29m57.075731536s) [noise=true , address=104.36.82.41:4793 , piece_id=3GGZZAHHV6MFSQM43SIS75MYAAZ5AFMMRHAGTXQOHP2VZJYCCAUQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1asDxxBjiU6yNbZ5c6uBmSWQ4QbVs5J3ErLbpKBSwqaptpNhpn, address:104.36.82.41:4793): protocol: write tcp 159.223.143.39:34488->104.36.82.41:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294844121 (26h29m57.075777527s) [seg_pos={0 0}, upload=25273]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:21:41.317380829 (26h29m57.098314199s) [upload=25273]: Done uploading segments... Jul 28 00:21:41.317406796 (26h29m57.098340175s) [upload=25273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.025699891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration: 463.804696ms FAIL (2025-07-28 00:21:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.844236ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X...XX.......] duration: 444.627472ms FAIL (2025-07-28 00:21:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.658704ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 295.437752ms FAIL (2025-07-28 00:20:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 295.464777ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)