Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX.............] duration: 16.947059392s FAIL (2025-07-27 00:48:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 16.947094079s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...............X...X.............] duration: 14.913075743s FAIL (2025-07-27 00:48:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.913100657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.......................X....] duration: 263.747187ms FAIL (2025-07-27 00:48:14 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 263.801914ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X...X.................XX....X......XX.XXXX...X.............] duration: 563.628456ms FAIL (2025-07-27 00:48:08 UTC) orbiter warning: context deadline exceeded Duration: 563.649916ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................X...............X...X............] duration: 642.625867ms FAIL (2025-07-27 00:47:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 642.662347ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX............] duration: 528.832363ms FAIL (2025-07-27 00:47:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.86917ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X.......................X...] duration: 208.586703ms FAIL (2025-07-27 00:47:41 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 208.616682ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X...X.................XX....X......XX.XXXX...X............] duration: 559.983986ms FAIL (2025-07-27 00:47:35 UTC) orbiter warning: context deadline exceeded Duration: 560.013933ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 935.846362ms FAIL (2025-07-27 00:47:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 935.884645ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.......................X..] duration: 234.402482ms FAIL (2025-07-27 00:47:15 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 234.436113ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 32.643311783s FAIL (2025-07-27 00:47:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 32.643365377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X......] duration: 505.07713ms FAIL (2025-07-27 00:47:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.108046ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................X...............X...X...........] duration: 14.242646982s FAIL (2025-07-27 00:47:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.242681586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX...........] duration: 522.492014ms FAIL (2025-07-27 00:46:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.532608ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.......................X.] duration: 222.55223ms FAIL (2025-07-27 00:46:47 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 222.589666ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X...X.................XX....X......XX.XXXX...X...........] duration: 565.194429ms FAIL (2025-07-27 00:46:38 UTC) orbiter warning: context deadline exceeded Duration: 565.311919ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X.....] duration: 592.374539ms FAIL (2025-07-27 00:46:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 592.41135ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................X...............X...X..........] duration: 6.478160051s FAIL (2025-07-27 00:46:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.478203885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..........] duration: 521.41018ms FAIL (2025-07-27 00:46:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.445056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 1.312270382s FAIL (2025-07-27 00:46:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.312301581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX.........] duration: 1.042112449s FAIL (2025-07-27 00:46:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.042165427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 1.523439713s FAIL (2025-07-27 00:45:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.523475283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................X...............X...X.........] duration: 676.064166ms FAIL (2025-07-27 00:45:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 676.086829ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.......................X] duration: 181.296028ms FAIL (2025-07-27 00:45:47 UTC) orbiter warning: uplink: invalid range 0:1024 (size:1024) Duration: 181.343122ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX........] duration: 603.015694ms FAIL (2025-07-27 00:45:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 603.050388ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 1.092593422s FAIL (2025-07-27 00:45:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.092624528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...............X...X........] duration: 5.450541394s FAIL (2025-07-27 00:45:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.450569311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X...X.................XX....X......XX.XXXX...X..........] duration: 603.613035ms FAIL (2025-07-27 00:45:27 UTC) orbiter warning: context deadline exceeded Duration: 603.634363ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X....] duration: 649.796949ms FAIL (2025-07-27 00:45:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 649.83338ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....X...X.................XX....X......XX.XXXX...X.........] duration: 832.541303ms FAIL (2025-07-27 00:45:04 UTC) orbiter warning: context deadline exceeded Duration: 832.57491ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X...] duration: 510.87209ms FAIL (2025-07-27 00:44:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.908599ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX.......] duration: 460.060342ms FAIL (2025-07-27 00:44:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.183266ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 214.62662ms FAIL (2025-07-27 00:44:33 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 214.657868ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 997.293914ms FAIL (2025-07-27 00:44:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 997.342779ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X...............................X...............X...X.......] duration: 835.615531ms FAIL (2025-07-27 00:44:26 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 835.643691ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX......] duration: 5.250631875s FAIL (2025-07-27 00:44:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.250669635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [.XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X..] duration: 545.030664ms FAIL (2025-07-27 00:44:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 545.062562ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 18.406910471s FAIL (2025-07-27 00:44:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 18.406943679s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XX...............................X...............X...X......] duration: 16.861179777s FAIL (2025-07-27 00:44:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.861210528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX....X...X.................XX....X......XX.XXXX...X........] duration: 444.840778ms FAIL (2025-07-27 00:43:53 UTC) orbiter warning: context deadline exceeded Duration: 444.863052ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 219.434599ms FAIL (2025-07-27 00:43:50 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 219.464618ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.XX...............................X...............X...X.....] duration: 14.357061061s FAIL (2025-07-27 00:43:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.35717252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [..XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X.] duration: 579.552266ms FAIL (2025-07-27 00:43:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 579.586412ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX.....] duration: 1.793800994s FAIL (2025-07-27 00:43:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.793853778s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX....X...X.................XX....X......XX.XXXX...X.......] duration: 564.178535ms FAIL (2025-07-27 00:43:15 UTC) orbiter warning: context deadline exceeded Duration: 564.212486ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX....] duration: 766.462297ms FAIL (2025-07-27 00:43:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 766.501527ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX...] duration: 15.784150983s FAIL (2025-07-27 00:43:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.784199256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 8.035511772s FAIL (2025-07-27 00:42:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 8.035539128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.X] duration: 30.054317038s FAIL (2025-07-27 00:42:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.054344311s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 195.672633ms FAIL (2025-07-27 00:42:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 195.706683ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)