Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 40.773827277s FAIL (2025-07-26 22:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.773862811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................X..X........XXX.XXX.X.XX..X..X......X...] duration: 472.802858ms FAIL (2025-07-26 22:21:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.836773ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................XX..XXX..X..XX..X..XX.X....X....X....] duration: 410.554499ms FAIL (2025-07-26 22:20:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.606838ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................X..X........XXX.XXX.X.XX..X..X......X..] duration: 508.800386ms FAIL (2025-07-26 22:20:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 508.845979ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 572.886484ms FAIL (2025-07-26 22:20:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.986166ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 1m0.040803194s FAIL (2025-07-26 22:20:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ]: Uploading piece... Jul 26 22:19:27.235986107 (27m43.016919488s) [noise=true , address=104.36.82.41:4778 , piece_id=FLGNR7WX7GZNL5XLCCF233IDACX2ZJTCLHJDKZGR5EBE5DT5TPMA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.236061262 (27m43.016994639s) [noise=true , address=79.127.205.234:4617 , piece_id=RY75UR24SEU7POCQC4MMWJMUZVL7LVI3K4AN6OU6OD3FYBIM777A, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.237084512 (27m43.01801791s) [noise=true , address=104.36.82.55:4199 , piece_id=SCXPHH5MFWDKAS7YUQIRZWAI5TNMO4JOZYNPTYDEHEHTYJRWX2CA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.237388855 (27m43.018322242s) [noise=true , address=104.36.82.2:4744 , piece_id=E2DNQWIRGNAYEPLIRRSZZT4JHZSR3TSICEXG4VPFTTBUWGXY4SQQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.237849474 (27m43.018782864s) [noise=true , address=64.49.13.170:4718 , piece_id=ARVQ5VTCTT6MA5NRY4CSLEG2MZ3UAVFZYL3YSUR2WQUID5VXXPPA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238228951 (27m43.019162338s) [noise=true , address=77.90.31.11:4560 , piece_id=L7IJ2ZJRLTELRD4IBCNCOI5G3PTEBP7QEH767LB7HEJ6D7673YBQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238618860 (27m43.019552239s) [noise=true , address=104.36.82.52:4109 , piece_id=DFJQPIHGD4XJH6PEEJWBMRM6AJWACYLPXJIDGXZ6PYIZKHKRVN5Q, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238933310 (27m43.019866695s) [noise=true , address=45.91.116.14:4222 , piece_id=GFC3ZCOT5T4S7M6KOY2MAUXCCIMFPKCVUC6D5K6M5LGGJ6M4UK6Q, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238998088 (27m43.019931465s) [noise=true , address=104.36.82.56:4261 , piece_id=QPZDHSVKAZFTWZUXKWGGJ5HIESZYJ4UCA6SAEOZ4RASY45LL445Q, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.246839994 (27m43.027773382s) [noise=true , address=77.90.31.16:4754 , piece_id=EEZ2M3MPSJ54RPOXSNZ2LHH7J36BLVBZZLPMHHNCEZZ4XU3DE4PA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.247343875 (27m43.028277261s) [noise=true , address=64.49.13.155:4923 , piece_id=SBBVP4SU7SSWCMGEDTYSU53LRS3SOLBRSJHCDHPDAIPL4WDQGIIQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.262390738 (27m43.043324127s) [noise=true , address=104.36.82.6:4908 , piece_id=BX6KPDECSK72CMBOBZ4434NGJ76TDGJVSCTDFE3WKIQTP4RFRAFQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.265715820 (27m43.04664921s) [noise=true , address=104.36.82.50:4032 , piece_id=QIWESO6HLFAZ3LWATNRYPSCDAFN65IBJXW6ZHBPR3F4KWLHY7UNQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.269101909 (27m43.050035297s) [noise=true , address=104.36.82.47:4519 , piece_id=RAU7UWKJKPGRKHETYIZ6CWQV4KSJPULVXDKGXY24F2UIR2I6CMLA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.271831206 (27m43.052764594s) [noise=true , address=104.36.82.21:4476 , piece_id=SWYERXQJN4ATA4IYDWZWAV4SILBJDC2VZFQDHBD5W33HTOEYQVAQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.324992170 (27m43.105925549s) [upload=434]: Next returned nil segment Jul 26 22:19:27.325016561 (27m43.10594993s) [upload=434]: Waiting for error group managing segments... Jul 26 22:19:27.378650484 (27m43.159583847s) [noise=true , address=77.90.31.16:4754 , piece_id=EEZ2M3MPSJ54RPOXSNZ2LHH7J36BLVBZZLPMHHNCEZZ4XU3DE4PA, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:27.378759348 (27m43.159692731s) [noise=true , address=77.90.31.11:4560 , piece_id=L7IJ2ZJRLTELRD4IBCNCOI5G3PTEBP7QEH767LB7HEJ6D7673YBQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:27.935102012 (27m43.716035378s) [noise=true , address=79.127.219.48:4090 , piece_id=YSMZH7YAAXL2G7BZEYNWZI4HDRFYQAL2RHKIKFQ42HAGNX6GWJVQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:28.107941349 (27m43.888874731s) [noise=true , address=64.49.13.155:4923 , piece_id=SBBVP4SU7SSWCMGEDTYSU53LRS3SOLBRSJHCDHPDAIPL4WDQGIIQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:28.134829073 (27m43.915762465s) [noise=true , address=79.127.205.234:4617 , piece_id=RY75UR24SEU7POCQC4MMWJMUZVL7LVI3K4AN6OU6OD3FYBIM777A, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:29.426104769 (27m45.207038133s) [noise=true , address=64.49.13.170:4718 , piece_id=ARVQ5VTCTT6MA5NRY4CSLEG2MZ3UAVFZYL3YSUR2WQUID5VXXPPA, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:33.820352231 (27m49.601285608s) [noise=true , address=45.91.116.14:4222 , piece_id=GFC3ZCOT5T4S7M6KOY2MAUXCCIMFPKCVUC6D5K6M5LGGJ6M4UK6Q, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:20:18.653274869 (28m34.434208255s) [noise=true , address=104.36.82.6:4908 , piece_id=BX6KPDECSK72CMBOBZ4434NGJ76TDGJVSCTDFE3WKIQTP4RFRAFQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:20:27.169426341 (28m42.950359722s) [noise=true , address=104.36.82.47:4519 , piece_id=RAU7UWKJKPGRKHETYIZ6CWQV4KSJPULVXDKGXY24F2UIR2I6CMLA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: read tcp 159.223.143.39:37040->104.36.82.47:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.169630094 (28m42.950563473s) [noise=true , address=104.36.82.21:4476 , piece_id=SWYERXQJN4ATA4IYDWZWAV4SILBJDC2VZFQDHBD5W33HTOEYQVAQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12NKwRvjiEpY7DrbqFGaDHQ8no5YV4cUALnpsvQ8icUPoLvMYen, address:104.36.82.21:4476): protocol: write tcp 159.223.143.39:55340->104.36.82.21:4476: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.169768451 (28m42.950701836s) [noise=true , address=104.36.82.56:4261 , piece_id=QPZDHSVKAZFTWZUXKWGGJ5HIESZYJ4UCA6SAEOZ4RASY45LL445Q, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: read tcp 159.223.143.39:40862->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.169872049 (28m42.950805428s) [noise=true , address=104.36.82.50:4032 , piece_id=QIWESO6HLFAZ3LWATNRYPSCDAFN65IBJXW6ZHBPR3F4KWLHY7UNQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1zj1d7Yzk4Z5Wda3YF6UgR5oSEhABcf6HCWwmHWXaX1CbKYJdQ, address:104.36.82.50:4032): protocol: read tcp 159.223.143.39:58490->104.36.82.50:4032: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.170988366 (28m42.951921756s) [noise=true , address=104.36.82.53:4153 , piece_id=QZ7HXROJ2XTQPFL6I4WHHIX7EGM22MZDUSW4YSMNQRQXT77IN3GA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 159.223.143.39:43288->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171211780 (28m42.952145168s) [noise=true , address=104.36.82.46:4490 , piece_id=HI4BCBNOHZYRSFJIJVKA64EYS3UFWSANSK47D3NRALBSYHVDJVPA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: read tcp 159.223.143.39:39554->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171280229 (28m42.952213609s) [noise=true , address=104.36.82.41:4778 , piece_id=FLGNR7WX7GZNL5XLCCF233IDACX2ZJTCLHJDKZGR5EBE5DT5TPMA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: write tcp 159.223.143.39:38720->104.36.82.41:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171616785 (28m42.952550169s) [noise=true , address=104.36.82.55:4199 , piece_id=SCXPHH5MFWDKAS7YUQIRZWAI5TNMO4JOZYNPTYDEHEHTYJRWX2CA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 159.223.143.39:42032->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171682575 (28m42.952615965s) [noise=true , address=104.36.82.2:4744 , piece_id=E2DNQWIRGNAYEPLIRRSZZT4JHZSR3TSICEXG4VPFTTBUWGXY4SQQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:104.36.82.2:4744): protocol: write tcp 159.223.143.39:43896->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171956497 (28m42.952889883s) [noise=true , address=104.36.82.52:4109 , piece_id=DFJQPIHGD4XJH6PEEJWBMRM6AJWACYLPXJIDGXZ6PYIZKHKRVN5Q, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: write tcp 159.223.143.39:55594->104.36.82.52:4109: use of closed network connection; conn closed: op Jul 26 22:20:27.171983490 (28m42.952916881s) [seg_pos={0 0}, upload=434]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:20:27.196484578 (28m42.977417944s) [upload=434]: Done uploading segments... Jul 26 22:20:27.196520272 (28m42.977453652s) [upload=434]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.040837011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [......................X..X........XXX.XXX.X.XX..X..X......X.] duration: 630.902017ms FAIL (2025-07-26 22:20:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 630.927609ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................X..X........XXX.XXX.X.XX..X..X......X] duration: 30.085567921s FAIL (2025-07-26 22:20:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.085591667s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................XX..XXX..X..XX..X..XX.X....X....X...] duration: 454.357217ms FAIL (2025-07-26 22:19:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.400063ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 511.32734ms FAIL (2025-07-26 22:19:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.360827ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.173404495s FAIL (2025-07-26 22:19:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.173435831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 16.069552051s FAIL (2025-07-26 22:19:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.658933536 (26m17.439866923s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659377297 (26m17.440310683s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659425641 (26m17.440359043s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659891300 (26m17.440824687s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660027462 (26m17.44096085s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660516844 (26m17.441450237s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660882579 (26m17.441815965s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660990717 (26m17.441924107s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661480854 (26m17.442414245s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661498728 (26m17.442432116s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661930324 (26m17.44286375s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662077993 (26m17.443011373s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662543009 (26m17.443476396s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.664341220 (26m17.445274608s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.772808761 (26m17.55374215s) [upload=408]: Next returned nil segment Jul 26 22:18:01.772831167 (26m17.553764552s) [upload=408]: Waiting for error group managing segments... Jul 26 22:18:01.799611104 (26m17.580544483s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:01.819792373 (26m17.600725758s) [noise=true , address=77.90.31.13:4658 , piece_id=ATJN5WMNODMLTCPGDPHFNOH7BM7PSH75F5GXCC7EXNBFWZWA4GEQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.282782349 (26m18.063715732s) [noise=true , address=79.127.205.235:4688 , piece_id=WGW73TAYOCEWTHBPJTNOF6NDSKHG7AZZ2W6WOWU4KASYV4MM5W5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.376301883 (26m18.157235262s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:04.945231567 (26m20.72616493s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:05.987660817 (26m21.768594198s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:07.487167062 (26m23.268100445s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:19:01.591229652 (27m17.37216303s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: write tcp 159.223.143.39:40798->104.36.82.14:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591410483 (27m17.372343861s) [noise=true , address=104.36.82.50:4041 , piece_id=MNTR6WWQAF7YTJWXT3AJKLCEGJE3REYN5H2SBNCGLMZPKEC5XYSA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12HSwcUKtymKKFJYd9acgwgQym2owswGXG8h61HAWdP7uooFCfk, address:104.36.82.50:4041): protocol: write tcp 159.223.143.39:44652->104.36.82.50:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591467344 (27m17.372400722s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:104.36.82.17:4362): protocol: write tcp 159.223.143.39:42742->104.36.82.17:4362: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591745171 (27m17.37267855s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591824050 (27m17.372757428s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:39950->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591898010 (27m17.372831391s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59, address:104.36.82.41:4788): protocol: write tcp 159.223.143.39:33814->104.36.82.41:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591966690 (27m17.372900068s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: write tcp 159.223.143.39:35654->104.36.82.53:4143: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592038412 (27m17.37297179s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 159.223.143.39:50510->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592113478 (27m17.37304686s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 159.223.143.39:58296->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592380636 (27m17.37331402s) [noise=true , address=104.36.82.47:4528 , piece_id=RUUKRJW5LIZSO3MSDQYXN6HIWNGIHCBZZTIHJGH2IU75Y43MMTOQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: read tcp 159.223.143.39:46510->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592666167 (27m17.373599555s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:01.592698834 (27m17.373632223s) [seg_pos={0 0}, upload=408]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:19:01.617673696 (27m17.398607076s) [upload=408]: Done uploading segments... Jul 26 22:19:01.617726168 (27m17.398659549s) [upload=408]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 16.069667095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................XX..XXX..X..XX..X..XX.X....X....X..] duration: 624.608196ms FAIL (2025-07-26 22:19:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 624.689159ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 648.888329ms FAIL (2025-07-26 22:19:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.921931ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031094754s FAIL (2025-07-26 22:19:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.658933536 (26m17.439866923s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659377297 (26m17.440310683s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659425641 (26m17.440359043s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659891300 (26m17.440824687s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660027462 (26m17.44096085s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660516844 (26m17.441450237s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660882579 (26m17.441815965s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660990717 (26m17.441924107s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661480854 (26m17.442414245s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661498728 (26m17.442432116s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661930324 (26m17.44286375s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662077993 (26m17.443011373s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662543009 (26m17.443476396s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.664341220 (26m17.445274608s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.772808761 (26m17.55374215s) [upload=408]: Next returned nil segment Jul 26 22:18:01.772831167 (26m17.553764552s) [upload=408]: Waiting for error group managing segments... Jul 26 22:18:01.799611104 (26m17.580544483s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:01.819792373 (26m17.600725758s) [noise=true , address=77.90.31.13:4658 , piece_id=ATJN5WMNODMLTCPGDPHFNOH7BM7PSH75F5GXCC7EXNBFWZWA4GEQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.282782349 (26m18.063715732s) [noise=true , address=79.127.205.235:4688 , piece_id=WGW73TAYOCEWTHBPJTNOF6NDSKHG7AZZ2W6WOWU4KASYV4MM5W5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.376301883 (26m18.157235262s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:04.945231567 (26m20.72616493s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:05.987660817 (26m21.768594198s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:07.487167062 (26m23.268100445s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:19:01.591229652 (27m17.37216303s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: write tcp 159.223.143.39:40798->104.36.82.14:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591410483 (27m17.372343861s) [noise=true , address=104.36.82.50:4041 , piece_id=MNTR6WWQAF7YTJWXT3AJKLCEGJE3REYN5H2SBNCGLMZPKEC5XYSA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12HSwcUKtymKKFJYd9acgwgQym2owswGXG8h61HAWdP7uooFCfk, address:104.36.82.50:4041): protocol: write tcp 159.223.143.39:44652->104.36.82.50:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591467344 (27m17.372400722s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:104.36.82.17:4362): protocol: write tcp 159.223.143.39:42742->104.36.82.17:4362: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591745171 (27m17.37267855s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591824050 (27m17.372757428s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:39950->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591898010 (27m17.372831391s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59, address:104.36.82.41:4788): protocol: write tcp 159.223.143.39:33814->104.36.82.41:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591966690 (27m17.372900068s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: write tcp 159.223.143.39:35654->104.36.82.53:4143: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592038412 (27m17.37297179s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 159.223.143.39:50510->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592113478 (27m17.37304686s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 159.223.143.39:58296->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592380636 (27m17.37331402s) [noise=true , address=104.36.82.47:4528 , piece_id=RUUKRJW5LIZSO3MSDQYXN6HIWNGIHCBZZTIHJGH2IU75Y43MMTOQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: read tcp 159.223.143.39:46510->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592666167 (27m17.373599555s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:01.592698834 (27m17.373632223s) [seg_pos={0 0}, upload=408]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:19:01.617673696 (27m17.398607076s) [upload=408]: Done uploading segments... Jul 26 22:19:01.617726168 (27m17.398659549s) [upload=408]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.031123277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................X..X........XXX.XXX.X.XX..X..X......] duration: 438.589903ms FAIL (2025-07-26 22:18:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 438.620741ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 11.204126513s FAIL (2025-07-26 22:18:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.204160093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................XX..XXX..X..XX..X..XX.X....X....X.] duration: 644.954476ms FAIL (2025-07-26 22:18:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 644.990413ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 575.763603ms FAIL (2025-07-26 22:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 575.787183ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................X..X........XXX.XXX.X.XX..X..X.....] duration: 542.317189ms FAIL (2025-07-26 22:18:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 542.344242ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.159094475s FAIL (2025-07-26 22:18:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.159115677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026301158s FAIL (2025-07-26 22:18:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 7N75M3XVA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.607834416 (25m17.388767809s) [noise=true , address=104.36.82.46:4511 , piece_id=OT66I7RL4BT44W26OYKS2DASSV3LXWE6IOS63SQ4ITD7JKEWV2IQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.608368825 (25m17.389302212s) [noise=true , address=104.36.82.20:4463 , piece_id=3LDMRO5O3F67KMXHUSLB2EM7P4OJMZSSQ32VZE32JINXCW27B3SQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.608765398 (25m17.38969878s) [noise=true , address=64.49.13.138:4482 , piece_id=RKK4PJBB24EFLE34Z3DACTF257T63JQARFS236ABEZ2ARBQ4E3WA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.609129021 (25m17.390062408s) [noise=true , address=104.36.82.21:4478 , piece_id=PZPSO6GX4CCMH6PXGV4OKCCAIRO2MZHSAEA4WO2JBOK334PKNXQQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.609503138 (25m17.390436525s) [noise=true , address=104.36.82.52:4114 , piece_id=7WXWHU57GSV54K2XCVUIRB6ZQJGIQV66A4MPCGWCDXWGZW27W2EA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.609856609 (25m17.390789997s) [noise=true , address=79.127.219.48:4090 , piece_id=24D7BBOVJWZBQECMHRA32MN5KMQMDF7HTQUPAOF7D6R5DGZD4QFQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.610205593 (25m17.391138979s) [noise=true , address=77.90.31.19:4875 , piece_id=YR5FX3GGFVCNS26Q7Y4XKWEHMU3RM2A4UVAEKACROJTBGZFJUXOA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.610610170 (25m17.39154355s) [noise=true , address=79.127.205.234:4610 , piece_id=SR7LZ4IUYRWBN6VSIF7W6KPQ7C2OU33JAZDPOT3STLPLZOBG6JHQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.611006218 (25m17.391939631s) [noise=true , address=79.127.201.215:4347 , piece_id=IAEEJE4VICZX7F3TIECDS5W3PHQ2V3KYWH4K42O7XL333AG4XVOA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.611425825 (25m17.39235921s) [noise=true , address=79.127.219.38:4007 , piece_id=CM26N5VO3VUP4XJTYWEGQWT3NLSXQFXXJFLNJUBDXVZVXUHNE65Q, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.611872771 (25m17.392806157s) [noise=true , address=104.36.82.47:4542 , piece_id=4LDMBNARMIDGDTZYSZ26DBUINO4KL4ZFEZ5BUNSMSFRI5IMKWYIA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.612281272 (25m17.39321466s) [noise=true , address=104.36.82.42:4807 , piece_id=PIAHLL22ZEXBV2LQTFMHSYSH57UZQQ3HZB3KT6AEOEZ5BWD6KEQQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.612682966 (25m17.393616345s) [noise=true , address=104.36.82.18:4382 , piece_id=S5MHPZ2MD2GIUWTGQLKEGO6KRA6A5UFKHBNLPTFZ7Q2KQJ7EG63Q, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.613057067 (25m17.393990444s) [noise=true , address=104.36.82.56:4233 , piece_id=Y3T2XHBZZ7E36QHCBPYZ2J7USGNXDJPBXYMULWBT7S326OHYQEAA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.613471934 (25m17.394405306s) [noise=true , address=104.36.82.53:4130 , piece_id=HKKR4UI5LX6VQT6KJJZ7EFILRC2YZX7VES3CJ3S7WMCX4KQBVHCA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.613868892 (25m17.394802278s) [noise=true , address=104.36.82.55:4201 , piece_id=V4CJF6J7W76DQ7CPA52RPQDZ4LNC54G5QPMYDN4HCK2U2OMV5D6Q, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.714531762 (25m17.495465156s) [upload=392]: Next returned nil segment Jul 26 22:17:01.714556162 (25m17.495489552s) [upload=392]: Waiting for error group managing segments... Jul 26 22:17:01.730358742 (25m17.511292111s) [noise=true , address=79.127.219.38:4007 , piece_id=CM26N5VO3VUP4XJTYWEGQWT3NLSXQFXXJFLNJUBDXVZVXUHNE65Q, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:01.756445212 (25m17.537378598s) [noise=true , address=77.90.31.19:4875 , piece_id=YR5FX3GGFVCNS26Q7Y4XKWEHMU3RM2A4UVAEKACROJTBGZFJUXOA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.297674681 (25m18.07860805s) [noise=true , address=79.127.205.234:4610 , piece_id=SR7LZ4IUYRWBN6VSIF7W6KPQ7C2OU33JAZDPOT3STLPLZOBG6JHQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.315687352 (25m18.096620738s) [noise=true , address=156.146.43.225:4448 , piece_id=UV57RX32D2SEDOFU6MQP4WNNMJJGTL2HG3L7NSSBUQCCSSOWORZQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:17:02.430379841 (25m18.211313219s) [noise=true , address=79.127.219.48:4090 , piece_id=24D7BBOVJWZBQECMHRA32MN5KMQMDF7HTQUPAOF7D6R5DGZD4QFQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.503180683 (25m18.284114058s) [noise=true , address=79.127.205.245:4247 , piece_id=G6C2GIU7MO7POFDYT4GDYXFXLQQCXXGU5FAU2E74VYX7N75M3XVA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.854260496 (25m18.635193885s) [noise=true , address=64.49.13.138:4482 , piece_id=RKK4PJBB24EFLE34Z3DACTF257T63JQARFS236ABEZ2ARBQ4E3WA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:03.254464117 (25m19.035397498s) [noise=true , address=79.127.201.215:4347 , piece_id=IAEEJE4VICZX7F3TIECDS5W3PHQ2V3KYWH4K42O7XL333AG4XVOA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:18:01.502194410 (26m17.283127794s) [noise=true , address=104.36.82.20:4463 , piece_id=3LDMRO5O3F67KMXHUSLB2EM7P4OJMZSSQ32VZE32JINXCW27B3SQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: rpc: dial tcp 104.36.82.20:4463: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502285807 (26m17.283219187s) [noise=true , address=104.36.82.21:4478 , piece_id=PZPSO6GX4CCMH6PXGV4OKCCAIRO2MZHSAEA4WO2JBOK334PKNXQQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:17kFjjJyvHApPUxUHJ6HeMEBCoo6xTbr8QizXjBEHwWj3mZUha, address:104.36.82.21:4478): protocol: rpc: dial tcp 104.36.82.21:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502347090 (26m17.283280466s) [noise=true , address=104.36.82.52:4114 , piece_id=7WXWHU57GSV54K2XCVUIRB6ZQJGIQV66A4MPCGWCDXWGZW27W2EA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: rpc: dial tcp 104.36.82.52:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502411717 (26m17.283345094s) [noise=true , address=104.36.82.42:4807 , piece_id=PIAHLL22ZEXBV2LQTFMHSYSH57UZQQ3HZB3KT6AEOEZ5BWD6KEQQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: rpc: dial tcp 104.36.82.42:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502456931 (26m17.283390309s) [noise=true , address=104.36.82.56:4233 , piece_id=Y3T2XHBZZ7E36QHCBPYZ2J7USGNXDJPBXYMULWBT7S326OHYQEAA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502506234 (26m17.283439611s) [noise=true , address=104.36.82.47:4542 , piece_id=4LDMBNARMIDGDTZYSZ26DBUINO4KL4ZFEZ5BUNSMSFRI5IMKWYIA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502550452 (26m17.283483828s) [noise=true , address=104.36.82.55:4201 , piece_id=V4CJF6J7W76DQ7CPA52RPQDZ4LNC54G5QPMYDN4HCK2U2OMV5D6Q, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502595223 (26m17.2835286s) [noise=true , address=104.36.82.18:4382 , piece_id=S5MHPZ2MD2GIUWTGQLKEGO6KRA6A5UFKHBNLPTFZ7Q2KQJ7EG63Q, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12S3SoJPMMyHpqDgG5Ry1KErPGHXugcTpikGNPjpwC7qq5sH9ho, address:104.36.82.18:4382): protocol: rpc: dial tcp 104.36.82.18:4382: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502691161 (26m17.28362454s) [noise=true , address=104.36.82.46:4511 , piece_id=OT66I7RL4BT44W26OYKS2DASSV3LXWE6IOS63SQ4ITD7JKEWV2IQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: rpc: dial tcp 104.36.82.46:4511: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.504258618 (26m17.285191985s) [noise=true , address=104.36.82.53:4130 , piece_id=HKKR4UI5LX6VQT6KJJZ7EFILRC2YZX7VES3CJ3S7WMCX4KQBVHCA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:18:01.504316278 (26m17.285249663s) [seg_pos={0 0}, upload=392]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:18:01.526127706 (26m17.307061086s) [upload=392]: Done uploading segments... Jul 26 22:18:01.526182580 (26m17.307115959s) [upload=392]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.026321347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 460.880534ms FAIL (2025-07-26 22:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.902254ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........................XX..XXX..X..XX..X..XX.X....X....X] duration: 30.031937519s FAIL (2025-07-26 22:17:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031963496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................X..X........XXX.XXX.X.XX..X..X....] duration: 2.448114829s FAIL (2025-07-26 22:17:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.448143588s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.88296956s FAIL (2025-07-26 22:17:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.883081671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033562229s FAIL (2025-07-26 22:17:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== ) [noise=true , address=104.36.82.53:4135 , piece_id=AMR4IJL6GIO2LWYKZO5NJ4IS3R45DYGLTDSNCKRI3JLP3YZV2ECA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.473745003 (24m17.25467839s) [noise=true , address=104.36.82.46:4514 , piece_id=LGVYOFWWLES44276UBMMALIL36Z3FJ3LOPTX5U2OGCHOG4E3B6DQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.473811231 (24m17.254744614s) [noise=true , address=104.36.82.56:4234 , piece_id=VNGTB2FLRW44V5ZE2TITOPNJYCRTQTGKPQVMUAKTTEZKCAWNZG2A, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.474250703 (24m17.255184091s) [noise=true , address=79.127.205.235:4682 , piece_id=EHTGCI7JSM642AATX23BVUFONM4APDHVIYMQAUNYLQT5MYGUZWIA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.474710580 (24m17.255643971s) [noise=true , address=64.49.13.138:4482 , piece_id=LTL6VLIEMLE7HIZXW2JRDXZFJ3VKCRLUECYKJIAT6KDRO6XTLABQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.474935918 (24m17.255869323s) [noise=true , address=79.127.201.215:4347 , piece_id=ODMZNTUSZCQHBR7SFU7PX3I2FE66WOBKOOKCMZ637QVVGYUKDG5Q, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.475565929 (24m17.256499317s) [noise=true , address=104.36.82.55:4203 , piece_id=QSGEUM7G42AUEG5JQ6CBAZQCNYNKEOL6TOX7CB5YF2ID4H6I6ICA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.475673696 (24m17.256607084s) [noise=true , address=104.36.82.11:4138 , piece_id=6HG5FAUP4TNT3DF2E2MOAEDD22LKWM7XQMJ6JUJOFAPEPAQMQKMA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.476048275 (24m17.256981665s) [noise=true , address=104.36.82.13:4219 , piece_id=SUUXCBWMZ3ZIVQCNBLZU33XZSTKYBDJPNYF7ATZ2ABTFBYPLZEUA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.476220112 (24m17.257153506s) [noise=true , address=79.127.219.38:4004 , piece_id=POKGXH5KKLJCSZYC5FZAX3PNM5KCQMINRP5XDPEWBYALQHCDP6NQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.476972597 (24m17.257905985s) [noise=true , address=104.36.82.17:4355 , piece_id=YATOCGA6ZHIJ4DMBBJIUWGYIJP2UZSZ76RXYGAZQT2UAEBXITVIA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.477444413 (24m17.258377803s) [noise=true , address=104.36.82.10:4080 , piece_id=4XZAMSGU4QDZSSQOFC5D6EDRLXUWTEPPMUDOUB5INXOIY5CO6G5Q, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.478256019 (24m17.259189397s) [noise=true , address=104.36.82.52:4115 , piece_id=TQSWEH44YYCIQA2TXJ7VECTWQPW3NKZ4ELII5RFVXATKUEQJLTRQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.478611307 (24m17.259544693s) [noise=true , address=77.90.31.18:4821 , piece_id=WPM7KD6YIQG5SEHWZCNWJAE6KJDF7HXOKT7Q4XNULD3DPAY6QZVA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.589099580 (24m17.370032958s) [upload=378]: Next returned nil segment Jul 26 22:16:01.589117146 (24m17.370050524s) [upload=378]: Waiting for error group managing segments... Jul 26 22:16:01.701337336 (24m17.482270701s) [noise=true , address=77.90.31.18:4821 , piece_id=WPM7KD6YIQG5SEHWZCNWJAE6KJDF7HXOKT7Q4XNULD3DPAY6QZVA, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.087132899 (24m17.868066268s) [noise=true , address=79.127.205.233:4640 , piece_id=HLRZZNUM7SB5XIYZENTWDZ5TWX7DB5MFXOHPJCSFIGZUWX6APGHQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.115884906 (24m17.896818269s) [noise=true , address=79.127.219.38:4004 , piece_id=POKGXH5KKLJCSZYC5FZAX3PNM5KCQMINRP5XDPEWBYALQHCDP6NQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.132652514 (24m17.913585894s) [noise=true , address=79.127.205.235:4682 , piece_id=EHTGCI7JSM642AATX23BVUFONM4APDHVIYMQAUNYLQT5MYGUZWIA, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.154969447 (24m17.935902829s) [noise=true , address=64.49.13.138:4482 , piece_id=LTL6VLIEMLE7HIZXW2JRDXZFJ3VKCRLUECYKJIAT6KDRO6XTLABQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:08.512970892 (24m24.293904255s) [noise=true , address=79.127.201.215:4347 , piece_id=ODMZNTUSZCQHBR7SFU7PX3I2FE66WOBKOOKCMZ637QVVGYUKDG5Q, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:17:01.400336069 (25m17.181269454s) [noise=true , address=104.36.82.56:4234 , piece_id=VNGTB2FLRW44V5ZE2TITOPNJYCRTQTGKPQVMUAKTTEZKCAWNZG2A, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400434778 (25m17.181368155s) [noise=true , address=104.36.82.17:4355 , piece_id=YATOCGA6ZHIJ4DMBBJIUWGYIJP2UZSZ76RXYGAZQT2UAEBXITVIA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:104.36.82.17:4355): protocol: rpc: dial tcp 104.36.82.17:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400496417 (25m17.181429793s) [noise=true , address=104.36.82.11:4138 , piece_id=6HG5FAUP4TNT3DF2E2MOAEDD22LKWM7XQMJ6JUJOFAPEPAQMQKMA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:55266->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400802646 (25m17.181736034s) [noise=true , address=104.36.82.10:4080 , piece_id=4XZAMSGU4QDZSSQOFC5D6EDRLXUWTEPPMUDOUB5INXOIY5CO6G5Q, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3, address:104.36.82.10:4080): protocol: rpc: dial tcp 104.36.82.10:4080: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400941716 (25m17.181875104s) [noise=true , address=104.36.82.40:4974 , piece_id=XOWD7SGRJGLSUGSX45XC6JFT7IAS3233C4TCYREROUNBRPWIKMJQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:127knbpTBAt5v44Kbz5tYDYfZhmXPa8tBaV43emRXkmKh2E5XMp, address:104.36.82.40:4974): protocol: rpc: dial tcp 104.36.82.40:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401182656 (25m17.182116036s) [noise=true , address=104.36.82.52:4115 , piece_id=TQSWEH44YYCIQA2TXJ7VECTWQPW3NKZ4ELII5RFVXATKUEQJLTRQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401243751 (25m17.182177173s) [noise=true , address=104.36.82.18:4367 , piece_id=UWKE6ZL7I7GDZMCIKRDBOHQRYDFGLRLWHVZTLYWQACTWUAJWKPXA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:104.36.82.18:4367): protocol: rpc: dial tcp 104.36.82.18:4367: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401294101 (25m17.182227463s) [noise=true , address=104.36.82.55:4203 , piece_id=QSGEUM7G42AUEG5JQ6CBAZQCNYNKEOL6TOX7CB5YF2ID4H6I6ICA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:104.36.82.55:4203): protocol: write tcp 159.223.143.39:35722->104.36.82.55:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401346802 (25m17.182280187s) [noise=true , address=104.36.82.53:4135 , piece_id=AMR4IJL6GIO2LWYKZO5NJ4IS3R45DYGLTDSNCKRI3JLP3YZV2ECA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: rpc: dial tcp 104.36.82.53:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401361155 (25m17.182294527s) [noise=true , address=104.36.82.13:4219 , piece_id=SUUXCBWMZ3ZIVQCNBLZU33XZSTKYBDJPNYF7ATZ2ABTFBYPLZEUA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: rpc: dial tcp 104.36.82.13:4219: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401395307 (25m17.18232867s) [noise=true , address=104.36.82.46:4514 , piece_id=LGVYOFWWLES44276UBMMALIL36Z3FJ3LOPTX5U2OGCHOG4E3B6DQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: read tcp 159.223.143.39:52234->104.36.82.46:4514: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401434086 (25m17.18236747s) [noise=true , address=104.36.82.47:4533 , piece_id=N46TAALRQWAKFWWRDTNXL4JHD7LGN7TTFM2H5DXHWPKJUDFCVVBA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: read tcp 159.223.143.39:35992->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401502173 (25m17.182435673s) [seg_pos={0 0}, upload=378]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:17:01.430081766 (25m17.211015139s) [upload=378]: Done uploading segments... Jul 26 22:17:01.430115832 (25m17.211049218s) [upload=378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.033580672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................X..X........XXX.XXX.X.XX..X..X...] duration: 791.779054ms FAIL (2025-07-26 22:16:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 791.907737ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 502.811049ms FAIL (2025-07-26 22:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.837367ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................XX..XXX..X..XX..X..XX.X....X....] duration: 582.093851ms FAIL (2025-07-26 22:16:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 582.119293ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.82698333s FAIL (2025-07-26 22:16:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.827075894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 538.89825ms FAIL (2025-07-26 22:16:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.922738ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029747561s FAIL (2025-07-26 22:16:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 207 (23m17.161303601s) [noise=true , address=79.127.219.38:4003 , piece_id=SNEXFG5WYJKZQJKA7LO66H4QWXIFUDT7F2OA6YG7Q3GJ66AIH3QA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.381141949 (23m17.162075338s) [noise=true , address=104.36.82.52:4089 , piece_id=IKUESTCMKMPXG5RILFLEXMKALF3U3FAWX5YAQLCL7P3Z2UPPXTOQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.381747090 (23m17.162680477s) [noise=true , address=104.36.82.20:4450 , piece_id=GB7FD4JA6ALNHUU456H6DPRUWNFDOXVMQXXRI3TLVIQIUFNTIOPQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.383633106 (23m17.164566488s) [noise=true , address=104.36.82.56:4264 , piece_id=SYT2VME2PVBBDOKT7KU4XMCXCLYB7E6BLHDAZ7BGPCAME2X5RGGQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.384385318 (23m17.165318707s) [noise=true , address=104.36.82.53:4122 , piece_id=HTOBANSLEJWB3L2FPO2GJIEXKY72AO3CAWX7FP46CJJRQMMPRGSA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.384819879 (23m17.165753261s) [noise=true , address=104.36.82.18:4400 , piece_id=WSGJRSWA64GZ6YZ4RG3XMXGDWG6TFO4ABFQPFT2NI3ME5HTNWFPA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.385609781 (23m17.166543172s) [noise=true , address=79.127.219.51:4212 , piece_id=ZKMHBMSFWI3MU426VOIZJAFC7RHMHNI3U4PVM6QZVXBPH2TAPYFQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.386549424 (23m17.167482813s) [noise=true , address=77.90.31.10:4533 , piece_id=5UNYLWAYXB6JUP3N5AJFX5WZO5F6AW6ZEJQDMHYYEZ4FQP2JQUWA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.387624314 (23m17.168557702s) [noise=true , address=104.36.82.14:4243 , piece_id=VW54S5OCLNCHZ7QBJDOMUMHN5242D5IUTCGUHFT3JRBQGR5QVIRQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.388751749 (23m17.169685137s) [noise=true , address=104.36.82.49:4978 , piece_id=C7ZCMIBVWBWFVNQNICLNGFJ4YWKTM5FQISUJVLPTQZ5LR4465EGA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.389242140 (23m17.170175528s) [noise=true , address=79.127.205.246:4287 , piece_id=SCAWLDCW6VY4ZBNQLQEPHIJXFEHLKTCH65FISV2AC2M4YTGLEPWA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.535167746 (23m17.316101133s) [upload=363]: Next returned nil segment Jul 26 22:15:01.535191331 (23m17.31612472s) [upload=363]: Waiting for error group managing segments... Jul 26 22:15:01.588605141 (23m17.369538532s) [noise=true , address=77.90.31.10:4533 , piece_id=5UNYLWAYXB6JUP3N5AJFX5WZO5F6AW6ZEJQDMHYYEZ4FQP2JQUWA, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:01.657193776 (23m17.43812714s) [noise=true , address=79.127.219.38:4003 , piece_id=SNEXFG5WYJKZQJKA7LO66H4QWXIFUDT7F2OA6YG7Q3GJ66AIH3QA, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:01.834852009 (23m17.615785401s) [noise=true , address=79.127.219.51:4212 , piece_id=ZKMHBMSFWI3MU426VOIZJAFC7RHMHNI3U4PVM6QZVXBPH2TAPYFQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:02.284727994 (23m18.065661375s) [noise=true , address=79.127.205.246:4287 , piece_id=SCAWLDCW6VY4ZBNQLQEPHIJXFEHLKTCH65FISV2AC2M4YTGLEPWA, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:03.065418763 (23m18.846352146s) [noise=true , address=79.127.201.215:4356 , piece_id=RSOVA25MF5G6U2HB32SUFDJPL7KUFE6KGTDB4ESYKO363AQ5QL4A, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:16:01.278094005 (24m17.059027385s) [noise=true , address=104.36.82.20:4471 , piece_id=HL3ESI753MHCYS5UKGG7GOC62ATLVVNIXG7T3WSC45Y5WQFAPRNA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12DMRnzj9hpP9tRn5DUz9WcU3N4jEZ3CZ9saVTFa1tefjDNs72r, address:104.36.82.20:4471): protocol: write tcp 159.223.143.39:57490->104.36.82.20:4471: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278398771 (24m17.059332147s) [noise=true , address=104.36.82.20:4450 , piece_id=GB7FD4JA6ALNHUU456H6DPRUWNFDOXVMQXXRI3TLVIQIUFNTIOPQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12UeL1LySKQySa8wjvG4FBs69cE1iWkW4Diz83nu1PxBCKxyvki, address:104.36.82.20:4450): protocol: write tcp 159.223.143.39:57250->104.36.82.20:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278755728 (24m17.059689106s) [noise=true , address=104.36.82.50:4013 , piece_id=U4X5MP4H7VTYTLXJNCHOEOF6NEJGNOZCSH7HCS4SGYIMIQBX2AAQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: write tcp 159.223.143.39:40656->104.36.82.50:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278845053 (24m17.059778433s) [noise=true , address=104.36.82.22:4531 , piece_id=OM3BCN4DZGLNN6OE6IVHF2BUILUDTIOFIAQFTHUF4WEETVJMWKRQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12jGenn3Fh2VB36jVPzyy2LxJgbLraXxL2yobw9tsBJLSDPJycN, address:104.36.82.22:4531): protocol: write tcp 159.223.143.39:52914->104.36.82.22:4531: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278937528 (24m17.059870906s) [noise=true , address=104.36.82.52:4089 , piece_id=IKUESTCMKMPXG5RILFLEXMKALF3U3FAWX5YAQLCL7P3Z2UPPXTOQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 159.223.143.39:35100->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278994899 (24m17.059928266s) [noise=true , address=104.36.82.53:4122 , piece_id=HTOBANSLEJWB3L2FPO2GJIEXKY72AO3CAWX7FP46CJJRQMMPRGSA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 159.223.143.39:43732->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279108733 (24m17.060042099s) [noise=true , address=104.36.82.51:4072 , piece_id=KH7NH6WNGKLYZCEOKZL6XRGWPF3NJ5IVCL7VSKM4MGGNW2RBEGSQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne, address:104.36.82.51:4072): protocol: write tcp 159.223.143.39:41306->104.36.82.51:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279178294 (24m17.060111669s) [noise=true , address=104.36.82.55:4196 , piece_id=TZ5D22G6OGPLQ6MP4QPCFU5F2L2KWDO6OXD3WGTNWNK463RENNKQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 159.223.143.39:49606->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279243926 (24m17.060177311s) [noise=true , address=104.36.82.14:4243 , piece_id=VW54S5OCLNCHZ7QBJDOMUMHN5242D5IUTCGUHFT3JRBQGR5QVIRQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:1FAn2HGJpXP5dfbYfL8PkxCfrBM3SHJVCASQMSy1kaRqLhyiRv, address:104.36.82.14:4243): protocol: write tcp 159.223.143.39:36398->104.36.82.14:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279270304 (24m17.060203674s) [noise=true , address=104.36.82.46:4515 , piece_id=UQMP35ZOJTLY6M7UU3SAKNT65O5B4XY5UQTNJQOS4BLREX4IPJOA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:52702->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279336827 (24m17.060270318s) [noise=true , address=104.36.82.56:4264 , piece_id=SYT2VME2PVBBDOKT7KU4XMCXCLYB7E6BLHDAZ7BGPCAME2X5RGGQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 159.223.143.39:33982->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279376952 (24m17.06031032s) [noise=true , address=104.36.82.49:4978 , piece_id=C7ZCMIBVWBWFVNQNICLNGFJ4YWKTM5FQISUJVLPTQZ5LR4465EGA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 159.223.143.39:50782->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279430818 (24m17.060364186s) [noise=true , address=104.36.82.18:4400 , piece_id=WSGJRSWA64GZ6YZ4RG3XMXGDWG6TFO4ABFQPFT2NI3ME5HTNWFPA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12UxA6DMP2dr2T8o352iih2D6UeEeGQBMm1SgPbHYHuWaSFdyYv, address:104.36.82.18:4400): protocol: write tcp 159.223.143.39:42226->104.36.82.18:4400: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279468321 (24m17.060401716s) [seg_pos={0 0}, upload=363]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:16:01.303336944 (24m17.084270327s) [upload=363]: Done uploading segments... Jul 26 22:16:01.303369884 (24m17.084303262s) [upload=363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029764521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................X..X........XXX.XXX.X.XX..X..X..] duration: 550.49436ms FAIL (2025-07-26 22:16:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.5572ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.853640355s FAIL (2025-07-26 22:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.85366664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XX..XXX..X..XX..X..XX.X....X...] duration: 516.6814ms FAIL (2025-07-26 22:15:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.899753ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................X..X........XXX.XXX.X.XX..X..X.] duration: 487.747512ms FAIL (2025-07-26 22:15:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.941243ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............................X..X........XXX.XXX.X.XX..X..X] duration: 30.048257977s FAIL (2025-07-26 22:15:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.04829145s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.00104872s FAIL (2025-07-26 22:15:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001077568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029585394s FAIL (2025-07-26 22:15:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 2 , piece_id=QJVOO3LHRH2IXFNRAE6RNSARSRUFK6SH76UMZJ3RIRVAT264IINA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.243680224 (22m17.024613608s) [noise=true , address=79.127.205.235:4682 , piece_id=LMXRIY4YQ6OQQYVTP722DDQP55XIUYN67YCDHCTCPQXTMF5GWRRQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.243932574 (22m17.024865942s) [noise=true , address=77.90.31.11:4579 , piece_id=5D5OF6UU25BJQA5FKWQJNQJOYTNNPYU56BYTHZVAHDAC5KUKMLFQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244087833 (22m17.02502121s) [noise=true , address=77.90.31.18:4822 , piece_id=JXKQPWVTHAYXEONUJHP3PHD7O324PJUJUAEL6OF37FKDNXE6TE6A, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244643513 (22m17.025576892s) [noise=true , address=79.127.201.215:4343 , piece_id=E33ZGZG4MW36CZE5IDGQJ3EXLK5DGBKO7I7VEQOZVCNNDOWA7GDA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244683583 (22m17.02561697s) [noise=true , address=45.91.116.14:4222 , piece_id=OK3B2RGFDUI42ETOQV3U5EQXVFUWBTPZK7REIPBRMPAJZSQZJLHQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244971930 (22m17.025905309s) [noise=true , address=104.36.82.13:4207 , piece_id=A37CBRYK2XUUBPGHZHZVFLNGABHNGKBTWPOSIR2WEOJIVE5AUXRA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245281092 (22m17.026214479s) [noise=true , address=104.36.82.46:4489 , piece_id=MQD4VSRFLEYUNJU2ON34M5X3MLYBZ24LO7UZ3MB5CIMQWGQOLLFQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245326951 (22m17.026260331s) [noise=true , address=104.36.82.53:4143 , piece_id=FOOCWCIO5C7TPZYLNJJFN6XGPWOG3OSE577PXHKUAO2BQ7T7ONXQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245664939 (22m17.026598318s) [noise=true , address=104.36.82.22:4536 , piece_id=IQGR6B6JAHA7BUVMKKNL6LPIWZBCZHRVFB52E7WOXJCPOL677VNQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245925159 (22m17.026858543s) [noise=true , address=104.36.82.55:4196 , piece_id=XLLCTG7QI2GIQZWYVUF7KBRV6VBVQGV2KLRC7ZM7G2BGFXU4DKPA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.246035744 (22m17.026969122s) [noise=true , address=104.36.82.47:4528 , piece_id=K7BHBZDPGFXERLSC4YFAVDS36AWL2WMGDXZGDVE4ITTK6QYG3JLQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.246558793 (22m17.027492185s) [noise=true , address=104.36.82.56:4243 , piece_id=TPPYUDVKOTAMAYRYLOVMOKWXTBIJDWJBXK5WYGQR3MGDU6HAHKRQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.247150043 (22m17.028083432s) [noise=true , address=77.90.31.16:4760 , piece_id=EP5RSDB75UZM6FDRJ7KY5RH5BU3UR74S6ED6VRURVIQPS3AK5I3A, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.247692056 (22m17.028625446s) [noise=true , address=79.127.205.234:4619 , piece_id=PBBI2Z5KWL2G2L6SYCFNKPB536DNWRTN45EKVZFXXX3RKITYY5DQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.248256110 (22m17.029189499s) [noise=true , address=104.36.82.44:4887 , piece_id=ULDZEFV2PWTG7VDMTVF6Q3ZAPDJ3TBIJU3EY37JY2NQFBGJMEMZQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.327424771 (22m17.108358198s) [upload=348]: Next returned nil segment Jul 26 22:14:01.327450970 (22m17.108384357s) [upload=348]: Waiting for error group managing segments... Jul 26 22:14:01.841386055 (22m17.622319433s) [noise=true , address=79.127.205.235:4682 , piece_id=LMXRIY4YQ6OQQYVTP722DDQP55XIUYN67YCDHCTCPQXTMF5GWRRQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:01.943651445 (22m17.724584815s) [noise=true , address=79.127.205.234:4619 , piece_id=PBBI2Z5KWL2G2L6SYCFNKPB536DNWRTN45EKVZFXXX3RKITYY5DQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:08.999820844 (22m24.780754242s) [noise=true , address=77.90.31.16:4760 , piece_id=EP5RSDB75UZM6FDRJ7KY5RH5BU3UR74S6ED6VRURVIQPS3AK5I3A, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:09.764776235 (22m25.545709601s) [noise=true , address=45.91.116.14:4222 , piece_id=OK3B2RGFDUI42ETOQV3U5EQXVFUWBTPZK7REIPBRMPAJZSQZJLHQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:10.344858287 (22m26.125791893s) [noise=true , address=79.127.201.215:4343 , piece_id=E33ZGZG4MW36CZE5IDGQJ3EXLK5DGBKO7I7VEQOZVCNNDOWA7GDA, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:13.090150330 (22m28.871083691s) [noise=true , address=77.90.31.18:4822 , piece_id=JXKQPWVTHAYXEONUJHP3PHD7O324PJUJUAEL6OF37FKDNXE6TE6A, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:14.048748563 (22m29.829681932s) [noise=true , address=64.49.13.138:4482 , piece_id=QJVOO3LHRH2IXFNRAE6RNSARSRUFK6SH76UMZJ3RIRVAT264IINA, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:15:01.141100547 (23m16.92203393s) [noise=true , address=104.36.82.2:4745 , piece_id=KTXAGLEJWDHSO5OCOG3XTNZ2YQER4VDG6OPPZQ7TEK2LYDDSB42A, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:1H9sLaFPDPZxKvwXho76wrqPiFe9yu87Ue9BT4ip92qfLmjbnF, address:104.36.82.2:4745): protocol: rpc: dial tcp 104.36.82.2:4745: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141424241 (23m16.922357622s) [noise=true , address=104.36.82.50:4025 , piece_id=O3RISI5REZAPP3YZH5XHM7BLTRYML3QQOVXXCLSHIKKJ5YXZYNTA, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:104.36.82.50:4025): protocol: rpc: dial tcp 104.36.82.50:4025: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141498271 (23m16.922431635s) [noise=true , address=104.36.82.13:4207 , piece_id=A37CBRYK2XUUBPGHZHZVFLNGABHNGKBTWPOSIR2WEOJIVE5AUXRA, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141587075 (23m16.922520453s) [noise=true , address=104.36.82.46:4489 , piece_id=MQD4VSRFLEYUNJU2ON34M5X3MLYBZ24LO7UZ3MB5CIMQWGQOLLFQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141653846 (23m16.922587224s) [noise=true , address=104.36.82.22:4536 , piece_id=IQGR6B6JAHA7BUVMKKNL6LPIWZBCZHRVFB52E7WOXJCPOL677VNQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12C2G1QGN37Wu55dBAxLiYWYAXmGapzjWUZUCKgCUbcjXcaRFmt, address:104.36.82.22:4536): protocol: write tcp 159.223.143.39:43194->104.36.82.22:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141713617 (23m16.922646994s) [noise=true , address=104.36.82.47:4528 , piece_id=K7BHBZDPGFXERLSC4YFAVDS36AWL2WMGDXZGDVE4ITTK6QYG3JLQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141735749 (23m16.922669127s) [noise=true , address=77.90.31.11:4579 , piece_id=5D5OF6UU25BJQA5FKWQJNQJOYTNNPYU56BYTHZVAHDAC5KUKMLFQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:15:01.141817910 (23m16.922751288s) [noise=true , address=104.36.82.56:4243 , piece_id=TPPYUDVKOTAMAYRYLOVMOKWXTBIJDWJBXK5WYGQR3MGDU6HAHKRQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141931334 (23m16.922864717s) [noise=true , address=104.36.82.53:4143 , piece_id=FOOCWCIO5C7TPZYLNJJFN6XGPWOG3OSE577PXHKUAO2BQ7T7ONXQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.142042923 (23m16.922976301s) [noise=true , address=104.36.82.55:4196 , piece_id=XLLCTG7QI2GIQZWYVUF7KBRV6VBVQGV2KLRC7ZM7G2BGFXU4DKPA, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.142091658 (23m16.923025036s) [noise=true , address=104.36.82.44:4887 , piece_id=ULDZEFV2PWTG7VDMTVF6Q3ZAPDJ3TBIJU3EY37JY2NQFBGJMEMZQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: write tcp 159.223.143.39:38072->104.36.82.44:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.142135504 (23m16.923068872s) [seg_pos={0 0}, upload=348]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:15:01.168270238 (23m16.949203616s) [upload=348]: Done uploading segments... Jul 26 22:15:01.168309637 (23m16.949243017s) [upload=348]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.029614945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............................XX..XXX..X..XX..X..XX.X....X..] duration: 434.521406ms FAIL (2025-07-26 22:14:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.553345ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 477.378606ms FAIL (2025-07-26 22:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.426298ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............................X..X........XXX.XXX.X.XX..X..] duration: 485.606987ms FAIL (2025-07-26 22:14:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.64932ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 516.806174ms FAIL (2025-07-26 22:14:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.828625ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 472.441641ms FAIL (2025-07-26 22:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.470558ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.0332824s FAIL (2025-07-26 22:13:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 48496s) [noise=true , address=104.36.82.42:4809 , piece_id=WLJONQZYSEDY4OGJ5KCQPF7JUWJPRWZTG7NUAQY4JB7D3R3IGGTA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.885564432 (21m4.666497818s) [noise=true , address=79.127.201.215:4350 , piece_id=Y7EARFRFLNNZQMSCSL27OJ5P6DH42DJBZ5TWO7G7QUPKQKB5P2RQ, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.885949517 (21m4.666882903s) [noise=true , address=104.36.82.3:4783 , piece_id=MGNS6XRLLLKUEWGJLPM55KEV4F73LLTRXNH7S7ZKHKXIVCLPFGWA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.886238666 (21m4.667172051s) [noise=true , address=77.90.31.11:4579 , piece_id=MUZRLA67GHCABWG7NOTN2WILNUEAFHEO2JCZFURG6H4P3DXCE4IA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.886331816 (21m4.667265202s) [noise=true , address=104.36.82.56:4251 , piece_id=JFJ33CH34YKRVXF2T3OWBX5NXU6NHCZMA7D7G6NITS4QR4A3OWTA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.886643846 (21m4.667577276s) [noise=true , address=79.127.205.234:4612 , piece_id=OPG2RRVGEYTEI733WQIX5B2V7RETEGR6UH53DNAPIAUJED5GFJXA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.887069399 (21m4.668002784s) [noise=true , address=64.49.13.169:4716 , piece_id=2SBENU3JIVLK7JP77U2DNPZ5J2G5DR4TXMC5DQVPTC74MHXH7PEA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.887462789 (21m4.668396167s) [noise=true , address=104.36.82.45:4916 , piece_id=XOUNTRQKZFOHHP23CYGUVP7N5V4MHZLKKHVHMNQDLIVHELWPYL3A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.887751055 (21m4.668684433s) [noise=true , address=79.127.205.245:4231 , piece_id=G7OUM6FPKEKC6J6FC6Y7273HGJSCT7TTDFW55KCZFUOTAEBTWEJA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.888068836 (21m4.669002223s) [noise=true , address=104.36.82.15:4289 , piece_id=JHOTWB7GBMIQAGNLYGVGPNNSAF2CRSO7POJTDOQ56VSF5H55E37A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.888347371 (21m4.669280756s) [noise=true , address=104.36.82.46:4513 , piece_id=VUGNBZAWSMCULWYJESRWAOWXGFHHXSI6FLMZVP44HFCW2EBBDR6A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.888730452 (21m4.669663839s) [noise=true , address=104.36.82.22:4527 , piece_id=BE3D4HBJ7WVUAMEOHU6VMJ6EHFWO3IMGEXEJ5OJGRSFRSP7BOSJQ, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889110640 (21m4.670044025s) [noise=true , address=79.127.205.233:4623 , piece_id=IMZL34MN2F4Z7YGCS3ITVX2D2CKPVPSS4PITRTK7VAAMJVER5SBA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889204917 (21m4.670138299s) [noise=true , address=104.36.82.53:4122 , piece_id=3LJMB7ICJWLEX7XWELI322DBRINF7ADI67OOOAW6TWKLIGS5SBPA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889503339 (21m4.670436715s) [noise=true , address=104.36.82.20:4459 , piece_id=PFILN2FPLX4EIK5GTIXDBD5LIDOC5X5KA67ZTQ75OI4B2SCYPC5A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889801820 (21m4.670735196s) [noise=true , address=77.90.31.16:4766 , piece_id=QFSKGC4HJRXDS3PXP2OGNZDE7LJEM2CTOPPXSLWJO2FL3HO47CKQ, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.961563148 (21m4.742496535s) [upload=331]: Next returned nil segment Jul 26 22:12:48.961584888 (21m4.742518278s) [upload=331]: Waiting for error group managing segments... Jul 26 22:12:49.001854366 (21m4.782787751s) [noise=true , address=77.90.31.16:4766 , piece_id=QFSKGC4HJRXDS3PXP2OGNZDE7LJEM2CTOPPXSLWJO2FL3HO47CKQ, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.005189349 (21m4.786122718s) [noise=true , address=77.90.31.11:4579 , piece_id=MUZRLA67GHCABWG7NOTN2WILNUEAFHEO2JCZFURG6H4P3DXCE4IA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.545610704 (21m5.326544076s) [noise=true , address=79.127.205.233:4623 , piece_id=IMZL34MN2F4Z7YGCS3ITVX2D2CKPVPSS4PITRTK7VAAMJVER5SBA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.593626741 (21m5.374560106s) [noise=true , address=79.127.205.245:4231 , piece_id=G7OUM6FPKEKC6J6FC6Y7273HGJSCT7TTDFW55KCZFUOTAEBTWEJA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.906649935 (21m5.687583311s) [noise=true , address=79.127.205.234:4612 , piece_id=OPG2RRVGEYTEI733WQIX5B2V7RETEGR6UH53DNAPIAUJED5GFJXA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:50.411069913 (21m6.192003281s) [noise=true , address=79.127.201.215:4350 , piece_id=Y7EARFRFLNNZQMSCSL27OJ5P6DH42DJBZ5TWO7G7QUPKQKB5P2RQ, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:50.433510940 (21m6.21444434s) [noise=true , address=64.49.13.169:4716 , piece_id=2SBENU3JIVLK7JP77U2DNPZ5J2G5DR4TXMC5DQVPTC74MHXH7PEA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:13:04.502088139 (21m20.283021504s) [noise=true , address=104.36.82.15:4289 , piece_id=JHOTWB7GBMIQAGNLYGVGPNNSAF2CRSO7POJTDOQ56VSF5H55E37A, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:13:48.817133201 (22m4.598066577s) [noise=true , address=104.36.82.56:4251 , piece_id=JFJ33CH34YKRVXF2T3OWBX5NXU6NHCZMA7D7G6NITS4QR4A3OWTA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817195068 (22m4.598128434s) [noise=true , address=104.36.82.11:4133 , piece_id=JM2TNQTKXJD4LSUTALXU4A66MMNDDVLH6HC5XZFVTKLDY7ZX4ZUA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817251552 (22m4.598184926s) [noise=true , address=104.36.82.42:4809 , piece_id=WLJONQZYSEDY4OGJ5KCQPF7JUWJPRWZTG7NUAQY4JB7D3R3IGGTA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1VZgucaa2gRuFhNxARMb7BDS6EGZXKChGPppbFksbPc5we7fTQ, address:104.36.82.42:4809): protocol: rpc: dial tcp 104.36.82.42:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817261989 (22m4.598195352s) [noise=true , address=104.36.82.22:4527 , piece_id=BE3D4HBJ7WVUAMEOHU6VMJ6EHFWO3IMGEXEJ5OJGRSFRSP7BOSJQ, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1nzM7fbo5JececsEvKgB7yyYXyjUrveZ7dVfS7FawJvP8DfgZA, address:104.36.82.22:4527): protocol: write tcp 159.223.143.39:52596->104.36.82.22:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817300004 (22m4.598233379s) [noise=true , address=104.36.82.46:4513 , piece_id=VUGNBZAWSMCULWYJESRWAOWXGFHHXSI6FLMZVP44HFCW2EBBDR6A, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817330388 (22m4.598263749s) [noise=true , address=104.36.82.53:4122 , piece_id=3LJMB7ICJWLEX7XWELI322DBRINF7ADI67OOOAW6TWKLIGS5SBPA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 159.223.143.39:54884->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817345019 (22m4.598278385s) [noise=true , address=104.36.82.3:4783 , piece_id=MGNS6XRLLLKUEWGJLPM55KEV4F73LLTRXNH7S7ZKHKXIVCLPFGWA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12ito72D7vikRdjK7BPt4LnhAK84bzi8AbZZPNWrTqmbjwQT1N, address:104.36.82.3:4783): protocol: write tcp 159.223.143.39:36370->104.36.82.3:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817388879 (22m4.598322241s) [noise=true , address=104.36.82.47:4536 , piece_id=QPKDGEP43676FQIFC47EKIP2TEEZUH2JF2Z6K4VR72MLC6COKSYA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 159.223.143.39:46980->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817400018 (22m4.598333383s) [noise=true , address=104.36.82.20:4459 , piece_id=PFILN2FPLX4EIK5GTIXDBD5LIDOC5X5KA67ZTQ75OI4B2SCYPC5A, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: write tcp 159.223.143.39:57166->104.36.82.20:4459: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817431070 (22m4.598364432s) [noise=true , address=104.36.82.45:4916 , piece_id=XOUNTRQKZFOHHP23CYGUVP7N5V4MHZLKKHVHMNQDLIVHELWPYL3A, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:104.36.82.45:4916): protocol: write tcp 159.223.143.39:40928->104.36.82.45:4916: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817465252 (22m4.598398636s) [seg_pos={0 0}, upload=331]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:13:48.847184845 (22m4.628118231s) [upload=331]: Done uploading segments... Jul 26 22:13:48.847215243 (22m4.628148617s) [upload=331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.033301136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................XX..XXX..X..XX..X..XX.X....X.] duration: 393.679809ms FAIL (2025-07-26 22:13:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 393.70479ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................X..X........XXX.XXX.X.XX..X.] duration: 878.079195ms FAIL (2025-07-26 22:13:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 878.101508ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [................................XX..XXX..X..XX..X..XX.X....X] duration: 30.028923705s FAIL (2025-07-26 22:13:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028958492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................X..X........XXX.XXX.X.XX..X] duration: 30.029171989s FAIL (2025-07-26 22:13:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02923937s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)