Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......XXX...XX.....X.XXX...............X..X......X.XX....X.] duration: 652.205884ms FAIL (2025-07-27 00:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 652.226519ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.......XXX...XX.....X.XXX...............X..X......X.XX....X] duration: 30.00041246s FAIL (2025-07-27 00:03:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000435887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX.] duration: 1.725330465s FAIL (2025-07-27 00:03:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.725372538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X.......X...X....XX..............X.X........X...XX........] duration: 36.095016896s FAIL (2025-07-27 00:03:32 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.095042213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.03146759s FAIL (2025-07-27 00:03:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.031522112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXX] duration: 30.031885123s FAIL (2025-07-27 00:03:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031910465s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032562196s FAIL (2025-07-27 00:02:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 3CZZYAGBID4AEN5ZYET6UGRX4CAPAQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.890168627 (2h10m13.671102013s) [noise=true , address=104.36.82.56:4254 , piece_id=CQFUNQLCXWVW3DAX57BLSKPSY7Y7ECR3VTNGEMEWSFYABBP7BJEA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.890647405 (2h10m13.671580792s) [noise=true , address=64.49.13.140:4508 , piece_id=MOT2SIO4D3RR5EDLTY2YMYTSGAKEWOHRPFAK4JXX2NUEZKXORGOQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.892500235 (2h10m13.67343362s) [noise=true , address=45.91.116.9:4277 , piece_id=HJEOOMIBFL7KIW4GK3G4KHQTAWAPIRCO3KFM33P4VTMLKVBDDL7Q, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.893017271 (2h10m13.673950655s) [noise=true , address=77.90.31.18:4833 , piece_id=ZWGOHSBPQWSXIQGG6QUYAXNJUQGFRWDCRRYXTPICZHH5QEJZXD4A, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.893432369 (2h10m13.674365751s) [noise=true , address=104.36.82.53:4141 , piece_id=S2K2VNMRALLQCWETY3E4ZO7M2PLGI4E3GHYMPHXYSIFZ2PWREEZQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.893844336 (2h10m13.674777721s) [noise=true , address=79.127.205.235:4657 , piece_id=26BBTLP4HWQ4P7TLLUFBR7PPR6JZU6DX73WP5JLJ7SWFPKKUUMLA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.894293428 (2h10m13.675226815s) [noise=true , address=77.90.31.17:4781 , piece_id=235LMHPJIKW422NJLWPFWML453IVPRSBQKHKHYUHGPDAQJJL3JVA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.894714611 (2h10m13.67564799s) [noise=true , address=104.36.82.46:4498 , piece_id=2Q5GHPRMO6ZN6SMWAL7JT7NMPPHISX56XJXCFFK466PIXWBWBCLQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.895164078 (2h10m13.676097465s) [noise=true , address=77.90.31.15:4732 , piece_id=UCCKTYWVWENHC5DU6KAISHSAHQ6ZJPBYCVQF55PC3PG35RS3TYVQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.895566819 (2h10m13.676500206s) [noise=true , address=104.36.82.55:4218 , piece_id=AA75RMM7UZ45ZLGEGTBX6477MSBAJWPGIJ2QMHU2MERTHPMWUROA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.895993151 (2h10m13.676926539s) [noise=true , address=79.127.201.215:4345 , piece_id=HLRWINXDAB7L7ILI22DU2E7DAYKKBF2NL254R6T4W4ABLVNXYCSQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.896414415 (2h10m13.6773478s) [noise=true , address=156.146.43.225:4449 , piece_id=BI4IEVSJU6DRU5CHVHLJ3GXYMUX42XXCQTXXDEBIMDVUGS5IU4HQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.899363215 (2h10m13.680296608s) [noise=true , address=104.36.82.47:4524 , piece_id=6SPF5NA2NNC6TPVHUPOMVYHLSHQDQ4NVG3FI5DCMHLNNRK6JUWZA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.899741795 (2h10m13.680675177s) [noise=true , address=104.36.82.54:4167 , piece_id=WJFNYIJHDLI5Q43WOAEKFPCNH6DSSEF3FTPGNTZ6QZT4PMKPWJ5A, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.968961593 (2h10m13.749894969s) [upload=1995]: Next returned nil segment Jul 27 00:01:57.968981228 (2h10m13.749914598s) [upload=1995]: Waiting for error group managing segments... Jul 27 00:01:58.000683759 (2h10m13.781617141s) [noise=true , address=77.90.31.18:4833 , piece_id=ZWGOHSBPQWSXIQGG6QUYAXNJUQGFRWDCRRYXTPICZHH5QEJZXD4A, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:01:58.041401918 (2h10m13.822335295s) [noise=true , address=156.146.43.225:4449 , piece_id=BI4IEVSJU6DRU5CHVHLJ3GXYMUX42XXCQTXXDEBIMDVUGS5IU4HQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 27 00:01:58.472609921 (2h10m14.253543298s) [noise=true , address=79.127.205.234:4620 , piece_id=M4YJIHRO4GY4EATF3MLB75GQT2QOWUNIP2Y5S5QP5IO42X4NY26Q, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:01:59.085103370 (2h10m14.866036737s) [noise=true , address=79.127.205.235:4657 , piece_id=26BBTLP4HWQ4P7TLLUFBR7PPR6JZU6DX73WP5JLJ7SWFPKKUUMLA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:00.633095575 (2h10m16.414028951s) [noise=true , address=79.127.220.97:4361 , piece_id=YXD5TXHRDT2M7U44FLXT6U3CZZYAGBID4AEN5ZYET6UGRX4CAPAQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:05.406614524 (2h10m21.187547892s) [noise=true , address=64.49.13.140:4508 , piece_id=MOT2SIO4D3RR5EDLTY2YMYTSGAKEWOHRPFAK4JXX2NUEZKXORGOQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:08.511226511 (2h10m24.292159889s) [noise=true , address=45.91.116.9:4277 , piece_id=HJEOOMIBFL7KIW4GK3G4KHQTAWAPIRCO3KFM33P4VTMLKVBDDL7Q, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:02:25.830189629 (2h10m41.611122998s) [noise=true , address=79.127.201.215:4345 , piece_id=HLRWINXDAB7L7ILI22DU2E7DAYKKBF2NL254R6T4W4ABLVNXYCSQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:57.818009349 (2h11m13.598942728s) [noise=true , address=104.36.82.53:4141 , piece_id=S2K2VNMRALLQCWETY3E4ZO7M2PLGI4E3GHYMPHXYSIFZ2PWREEZQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818131163 (2h11m13.599064541s) [noise=true , address=77.90.31.17:4781 , piece_id=235LMHPJIKW422NJLWPFWML453IVPRSBQKHKHYUHGPDAQJJL3JVA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT, address:77.90.31.17:4781): protocol: rpc: dial tcp 77.90.31.17:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818227960 (2h11m13.599161339s) [noise=true , address=104.36.82.55:4218 , piece_id=AA75RMM7UZ45ZLGEGTBX6477MSBAJWPGIJ2QMHU2MERTHPMWUROA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: rpc: dial tcp 104.36.82.55:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818303610 (2h11m13.599236987s) [noise=true , address=104.36.82.54:4167 , piece_id=WJFNYIJHDLI5Q43WOAEKFPCNH6DSSEF3FTPGNTZ6QZT4PMKPWJ5A, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: rpc: dial tcp 104.36.82.54:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818418934 (2h11m13.599352329s) [noise=true , address=104.36.82.56:4254 , piece_id=CQFUNQLCXWVW3DAX57BLSKPSY7Y7ECR3VTNGEMEWSFYABBP7BJEA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1k4x3HL3PNHwKJX83RyqvnsATHk3HmeMiC12A7raLN58wzvkUN, address:104.36.82.56:4254): protocol: write tcp 159.223.143.39:35910->104.36.82.56:4254: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818736110 (2h11m13.599669477s) [noise=true , address=104.36.82.35:4585 , piece_id=PEGX62F6H5HWLFCK6Z4MB3JL2452Z2EIMWWYNMQO54MEVKJAO6VQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:104.36.82.35:4585): protocol: rpc: dial tcp 104.36.82.35:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818826102 (2h11m13.599759481s) [noise=true , address=77.90.31.10:4523 , piece_id=FC4CXEWPO3RHMEP5ELANVLNZ22B4CVTMS5XOO5BLVFCRUOOYXUKA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: write tcp 159.223.143.39:52404->77.90.31.10:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818868442 (2h11m13.599801813s) [noise=true , address=104.36.82.46:4498 , piece_id=2Q5GHPRMO6ZN6SMWAL7JT7NMPPHISX56XJXCFFK466PIXWBWBCLQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: rpc: dial tcp 104.36.82.46:4498: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.819242911 (2h11m13.600176279s) [noise=true , address=104.36.82.47:4524 , piece_id=6SPF5NA2NNC6TPVHUPOMVYHLSHQDQ4NVG3FI5DCMHLNNRK6JUWZA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:57.819329587 (2h11m13.600262957s) [noise=true , address=77.90.31.15:4732 , piece_id=UCCKTYWVWENHC5DU6KAISHSAHQ6ZJPBYCVQF55PC3PG35RS3TYVQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:45896->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.819353836 (2h11m13.600287229s) [seg_pos={0 0}, upload=1995]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 27 00:02:57.847511553 (2h11m13.628444923s) [upload=1995]: Done uploading segments... Jul 27 00:02:57.847557732 (2h11m13.628491099s) [upload=1995]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.032577916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X..X.......X...X....XX..............X.X........X...XX.......] duration: 27.896911117s FAIL (2025-07-27 00:02:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.896935399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XX] duration: 30.033693002s FAIL (2025-07-27 00:02:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.033713717s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.030364293s FAIL (2025-07-27 00:02:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UA3LCNXW2F7YUJ233ZW5SSU7PFQ, seg_pos={0 0}, upload=1984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: rpc: dial tcp 104.36.82.5:4840: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:10.356708938 (2h9m26.137642303s) [seg_pos={0 0}, upload=1984]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:01:10.356794519 (2h9m26.137727884s) [upload=1984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:01:10.356799861 (2h9m26.137733227s) [upload=1984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:01:10.412842511 (2h9m26.193775897s) [upload=1984]: Done uploading segments... Jul 27 00:01:10.412860639 (2h9m26.193794016s) [upload=1984]: Upload finished. err: <nil> Jul 27 00:02:02.081104274 (2h10m17.862037643s) [upload=1997]: Starting upload Jul 27 00:02:02.081251072 (2h10m17.862184436s) [upload=1997]: Done starting upload Jul 27 00:02:02.081353965 (2h10m17.862287335s) [upload=1997]: Uploading segments... Jul 27 00:02:02.081859557 (2h10m17.862792937s) [upload=1997]: Got next segment. Inline: false Jul 27 00:02:02.081960739 (2h10m17.862894117s) [upload=1997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:02.081967317 (2h10m17.862900686s) [upload=1997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:02.109791587 (2h10m17.890724967s) [seg_pos={0 0}, upload=1997]: Begin upload segment... Jul 27 00:02:02.109836843 (2h10m17.89077021s) [seg_pos={0 0}, upload=1997]: Done begin upload segment. Jul 27 00:02:02.109861711 (2h10m17.89079509s) [noise=true , address=77.90.31.11:4554 , piece_id=CDON6KZTEAQ5HEPYUCPGKNIUUEDAPQQUJNMXPXVTJHNWHNR3SU6Q, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.109931120 (2h10m17.890864506s) [noise=true , address=104.36.82.12:4151 , piece_id=AKTJ2XIL57XAVIOKKBQ6VUIHGFUO3Y77ZTCIRPNFHPBW35ISXD2A, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110250765 (2h10m17.891184144s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110442379 (2h10m17.891375765s) [noise=true , address=104.36.82.13:4187 , piece_id=6XUZ6QLTGNLAOLJUKJEBMUGRZN4WXESMUGRK2TY2OVHS5P7JYJRA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110619463 (2h10m17.891552838s) [noise=true , address=77.90.31.11:4553 , piece_id=7ZCRLP2DECHLTM5BTCZ7SFAP5YGI2MTUMSMAV5PNQD6XV5IYXQ4A, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110872752 (2h10m17.891806134s) [noise=true , address=77.90.31.14:4663 , piece_id=FISRVFTJJ5EOMDT67CIY5TCUGJBRHQOSOSRSBAILRJ2MDW2W7FCA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111008596 (2h10m17.891941977s) [noise=true , address=45.91.116.9:4263 , piece_id=R23P77YGB3NWXJ55ZTZQB6IGLXZVFIQAMQHOPQ3CADP4F2IK3KFQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111325039 (2h10m17.892258417s) [noise=true , address=77.90.31.15:4698 , piece_id=3GAZYPEG6IEUDXD4B6EGIPEC7V5RI2QKOSCTARNT5FUH3ECMWLNA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111514542 (2h10m17.892447923s) [noise=true , address=45.91.116.12:4225 , piece_id=P7UCY4S4R3C7GGXWGKTB6PMT2T474KDJ2H3W7KRZYIEUBJX72XVQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111726483 (2h10m17.892659862s) [noise=true , address=45.91.116.7:4009 , piece_id=4WZJXUMT3EVUIYSL7UC5RV6X4RKIFFP47BGFLYBU3WWRIODVVURQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111844591 (2h10m17.892777976s) [noise=true , address=77.90.31.12:4590 , piece_id=N6HN4TWGPT7YINQDENR47W2FORXGG4PX2RZPKVWMJRIFOTW4SLLQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112257740 (2h10m17.893191145s) [noise=true , address=77.90.31.14:4662 , piece_id=ZFKUGFXUE4WMRAZKSTPFG3STZTN5TLLMZTZBPG65M4OAY2EGVO3Q, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112458788 (2h10m17.893392172s) [noise=true , address=45.91.116.7:4010 , piece_id=6EUSJ4W32V62WD4FKNG5Y6CJVPRQJF4ZCSAAEZPAODWI4TIN6VCA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112716595 (2h10m17.893649979s) [noise=true , address=45.91.116.10:4083 , piece_id=KZSTVSEJAU4PMREKD4ZZFHILVKBBAJMAUS5VDWK25VCDXC7USTEQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112841825 (2h10m17.893775207s) [noise=true , address=45.91.116.6:4973 , piece_id=TQVMQJWU7W2WKF5D7XESP677W2VU36MR35SPRPZDFLHC4KNSHPAQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.113114233 (2h10m17.894047617s) [noise=true , address=104.36.82.11:4115 , piece_id=BRHAQETHRLU6WA6OFRGYQCDTPAWBUTEVQHRDKR3J7ZOZBTI2OYSA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.225873984 (2h10m18.006807366s) [upload=1997]: Next returned nil segment Jul 27 00:02:02.225894604 (2h10m18.006827969s) [upload=1997]: Waiting for error group managing segments... Jul 27 00:02:02.228312435 (2h10m18.009245824s) [noise=true , address=45.91.116.7:4010 , piece_id=6EUSJ4W32V62WD4FKNG5Y6CJVPRQJF4ZCSAAEZPAODWI4TIN6VCA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.229102309 (2h10m18.010035689s) [noise=true , address=45.91.116.10:4083 , piece_id=KZSTVSEJAU4PMREKD4ZZFHILVKBBAJMAUS5VDWK25VCDXC7USTEQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.229791256 (2h10m18.010724638s) [noise=true , address=77.90.31.14:4662 , piece_id=ZFKUGFXUE4WMRAZKSTPFG3STZTN5TLLMZTZBPG65M4OAY2EGVO3Q, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.230406236 (2h10m18.011339609s) [noise=true , address=45.91.116.12:4225 , piece_id=P7UCY4S4R3C7GGXWGKTB6PMT2T474KDJ2H3W7KRZYIEUBJX72XVQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.231200277 (2h10m18.012133641s) [noise=true , address=77.90.31.11:4554 , piece_id=CDON6KZTEAQ5HEPYUCPGKNIUUEDAPQQUJNMXPXVTJHNWHNR3SU6Q, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.232286154 (2h10m18.013219539s) [noise=true , address=77.90.31.14:4663 , piece_id=FISRVFTJJ5EOMDT67CIY5TCUGJBRHQOSOSRSBAILRJ2MDW2W7FCA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.232986510 (2h10m18.013919889s) [noise=true , address=45.91.116.6:4973 , piece_id=TQVMQJWU7W2WKF5D7XESP677W2VU36MR35SPRPZDFLHC4KNSHPAQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.233587528 (2h10m18.014520896s) [noise=true , address=45.91.116.7:4009 , piece_id=4WZJXUMT3EVUIYSL7UC5RV6X4RKIFFP47BGFLYBU3WWRIODVVURQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.250677686 (2h10m18.031611064s) [noise=true , address=77.90.31.11:4553 , piece_id=7ZCRLP2DECHLTM5BTCZ7SFAP5YGI2MTUMSMAV5PNQD6XV5IYXQ4A, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.396939240 (2h10m18.177872604s) [noise=true , address=45.91.116.9:4263 , piece_id=R23P77YGB3NWXJ55ZTZQB6IGLXZVFIQAMQHOPQ3CADP4F2IK3KFQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:03.681602543 (2h10m19.462535911s) [noise=true , address=77.90.31.12:4590 , piece_id=N6HN4TWGPT7YINQDENR47W2FORXGG4PX2RZPKVWMJRIFOTW4SLLQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:32.048899053 (2h10m47.829832419s) [noise=true , address=104.36.82.12:4151 , piece_id=AKTJ2XIL57XAVIOKKBQ6VUIHGFUO3Y77ZTCIRPNFHPBW35ISXD2A, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.048963088 (2h10m47.829896454s) [noise=true , address=104.36.82.13:4187 , piece_id=6XUZ6QLTGNLAOLJUKJEBMUGRZN4WXESMUGRK2TY2OVHS5P7JYJRA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.049013589 (2h10m47.829946965s) [noise=true , address=77.90.31.15:4698 , piece_id=3GAZYPEG6IEUDXD4B6EGIPEC7V5RI2QKOSCTARNT5FUH3ECMWLNA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.049091588 (2h10m47.83002497s) [noise=true , address=104.36.82.11:4115 , piece_id=BRHAQETHRLU6WA6OFRGYQCDTPAWBUTEVQHRDKR3J7ZOZBTI2OYSA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030394104s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.......XXX...XX.....X.XXX...............X..X......X.XX....] duration: 570.926463ms FAIL (2025-07-27 00:02:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 570.953691ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...X] duration: 30.043460857s FAIL (2025-07-27 00:02:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.043489951s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X..X.......X...X....XX..............X.X........X...XX......] duration: 27.729529625s FAIL (2025-07-27 00:01:59 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.729551816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028539159s FAIL (2025-07-27 00:01:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== segment. Jul 27 00:00:57.775236716 (2h9m13.556170102s) [noise=true , address=77.90.31.13:4644 , piece_id=B7USLVEMOWW67S2V5ELBPFBYFQEXFYJQWSXSZXM2RTGY5PKDF6TQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.775268684 (2h9m13.55620207s) [noise=true , address=64.49.13.26:4953 , piece_id=TILKJGDG6YP7AZARW4NO2UHD6RNJHAPZROT6SWD2VLI53WQ7FY2Q, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.775733963 (2h9m13.55666735s) [noise=true , address=104.36.82.46:4493 , piece_id=SHKP3MLGB3A5ABL3EB3RAG25ORWRVGAIVYKJY4MYG3HAWU6JL6HQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.775739558 (2h9m13.556672936s) [noise=true , address=77.90.31.15:4713 , piece_id=WLHB7FUEDT3D2ORHQBKD42R2QLEM2IBHQP2OTLXAVOMPU5ZETTQA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.776244464 (2h9m13.55717785s) [noise=true , address=79.127.205.234:4610 , piece_id=UABIHF3TSXVKUXDIU4F6MDLY7UBKGUHW45R575QOOB5QTN2WUBXA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.776298606 (2h9m13.557231985s) [noise=true , address=104.36.82.47:4528 , piece_id=EZ7TXN3VVD4M2CXGW2RENPLHPQBXO2MHXI45UV3GI4IS5D26VNTQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.776753875 (2h9m13.55768726s) [noise=true , address=79.127.205.246:4272 , piece_id=WWZENSKQVVETRJTNNE7CDDC2I6M7UFPUJMC3WMX6OQFVVZXULENQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.777180560 (2h9m13.55811394s) [noise=true , address=104.36.82.52:4089 , piece_id=CAW3CLMSY3Q7RLZFWWRPWR3PUXNA7VZTPBBTLR2HN5LVQUR4DCSQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.780407316 (2h9m13.561340701s) [noise=true , address=104.36.82.53:4129 , piece_id=S64HIMTAERGHA6MM6MA5FPWA576GL4EO5V2PCHF2GOJAREOJCAGA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.780856964 (2h9m13.561790345s) [noise=true , address=104.36.82.56:4243 , piece_id=RS3BMQETGHRWURDAGH6EYJKQ3UXZRZWIVRLE3OR6NJILYVXZA2LA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.781209672 (2h9m13.56214305s) [noise=true , address=104.36.82.55:4209 , piece_id=UGUI6HGMIIDS2NHYFSFSEDTAWK5MK2MKVFTTYWVCW3G27A6H7DPQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.781525117 (2h9m13.562458496s) [noise=true , address=77.90.31.11:4581 , piece_id=UM26XZ4SIHWDAYNYPZ7MKGJ4XRDBPSAB55SPZU7SWI7GF5HRYMFA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.781890781 (2h9m13.56282416s) [noise=true , address=79.127.205.233:4643 , piece_id=7SUPFJWDQRRZORZTR5ZHK5IH47YI6JZNL2J3II3VJVQH2OAJS73Q, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.784119727 (2h9m13.565053105s) [noise=true , address=45.91.116.14:4206 , piece_id=R5ZWZGF4PQJBWY2BO4IWANJV27AWQYD3NS76DFMAEKGRYI6UFDBQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.785132722 (2h9m13.566066109s) [noise=true , address=79.127.219.38:4003 , piece_id=PG2TCCGX2URHK65ZGH6HGPL7DKGXF5L5TEVZBR3TKNCUJZL3TNRQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.785589584 (2h9m13.566522961s) [noise=true , address=64.49.13.138:4483 , piece_id=EA4VPG4UW22XVN3CH5QPOJ6RY2EYQV27FLORILQM26Q2WKNIYQNQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.785927870 (2h9m13.566861245s) [noise=true , address=77.90.31.16:4748 , piece_id=CZ5QM3M3RX27XVQVAOTIESGHKKSBY57HXSZCKZGAZ7KAIO3JARPQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.786322650 (2h9m13.567256031s) [noise=true , address=79.127.220.101:4041 , piece_id=G2WQS7UFY26I3UTCSQSYYH2DOEN4GMYQ5Y7U55ZW26YSV5KCX5DA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.859084590 (2h9m13.640017966s) [upload=1979]: Next returned nil segment Jul 27 00:00:57.859122614 (2h9m13.64005599s) [upload=1979]: Waiting for error group managing segments... Jul 27 00:00:57.863654302 (2h9m13.644587669s) [noise=true , address=45.91.116.14:4206 , piece_id=R5ZWZGF4PQJBWY2BO4IWANJV27AWQYD3NS76DFMAEKGRYI6UFDBQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:57.864360694 (2h9m13.645294073s) [noise=true , address=77.90.31.11:4581 , piece_id=UM26XZ4SIHWDAYNYPZ7MKGJ4XRDBPSAB55SPZU7SWI7GF5HRYMFA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:57.901398234 (2h9m13.682331611s) [noise=true , address=79.127.220.101:4041 , piece_id=G2WQS7UFY26I3UTCSQSYYH2DOEN4GMYQ5Y7U55ZW26YSV5KCX5DA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:57.929458781 (2h9m13.710392161s) [noise=true , address=77.90.31.13:4644 , piece_id=B7USLVEMOWW67S2V5ELBPFBYFQEXFYJQWSXSZXM2RTGY5PKDF6TQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:58.051130705 (2h9m13.832064077s) [noise=true , address=79.127.219.38:4003 , piece_id=PG2TCCGX2URHK65ZGH6HGPL7DKGXF5L5TEVZBR3TKNCUJZL3TNRQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:58.221274989 (2h9m14.002208364s) [noise=true , address=64.49.13.138:4483 , piece_id=EA4VPG4UW22XVN3CH5QPOJ6RY2EYQV27FLORILQM26Q2WKNIYQNQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:58.373358143 (2h9m14.154291517s) [noise=true , address=79.127.205.246:4272 , piece_id=WWZENSKQVVETRJTNNE7CDDC2I6M7UFPUJMC3WMX6OQFVVZXULENQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:58.381904170 (2h9m14.16283755s) [noise=true , address=79.127.205.233:4643 , piece_id=7SUPFJWDQRRZORZTR5ZHK5IH47YI6JZNL2J3II3VJVQH2OAJS73Q, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:59.087833210 (2h9m14.868766582s) [noise=true , address=79.127.205.234:4610 , piece_id=UABIHF3TSXVKUXDIU4F6MDLY7UBKGUHW45R575QOOB5QTN2WUBXA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:06.359428530 (2h9m22.140361934s) [noise=true , address=77.90.31.16:4748 , piece_id=CZ5QM3M3RX27XVQVAOTIESGHKKSBY57HXSZCKZGAZ7KAIO3JARPQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:09.278432011 (2h9m25.059365374s) [noise=true , address=64.49.13.26:4953 , piece_id=TILKJGDG6YP7AZARW4NO2UHD6RNJHAPZROT6SWD2VLI53WQ7FY2Q, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:11.859875333 (2h9m27.640808702s) [noise=true , address=77.90.31.15:4713 , piece_id=WLHB7FUEDT3D2ORHQBKD42R2QLEM2IBHQP2OTLXAVOMPU5ZETTQA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:57.706427957 (2h10m13.487361322s) [noise=true , address=104.36.82.55:4209 , piece_id=UGUI6HGMIIDS2NHYFSFSEDTAWK5MK2MKVFTTYWVCW3G27A6H7DPQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR, address:104.36.82.55:4209): protocol: write tcp 159.223.143.39:34528->104.36.82.55:4209: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706497977 (2h10m13.487431342s) [noise=true , address=104.36.82.47:4528 , piece_id=EZ7TXN3VVD4M2CXGW2RENPLHPQBXO2MHXI45UV3GI4IS5D26VNTQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 159.223.143.39:40502->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706550200 (2h10m13.487483568s) [noise=true , address=104.36.82.56:4243 , piece_id=RS3BMQETGHRWURDAGH6EYJKQ3UXZRZWIVRLE3OR6NJILYVXZA2LA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: write tcp 159.223.143.39:36696->104.36.82.56:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706601741 (2h10m13.48753511s) [noise=true , address=104.36.82.46:4493 , piece_id=SHKP3MLGB3A5ABL3EB3RAG25ORWRVGAIVYKJY4MYG3HAWU6JL6HQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 159.223.143.39:39626->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706648689 (2h10m13.487582065s) [noise=true , address=104.36.82.52:4089 , piece_id=CAW3CLMSY3Q7RLZFWWRPWR3PUXNA7VZTPBBTLR2HN5LVQUR4DCSQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 159.223.143.39:45398->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706693621 (2h10m13.487626988s) [noise=true , address=104.36.82.53:4129 , piece_id=S64HIMTAERGHA6MM6MA5FPWA576GL4EO5V2PCHF2GOJAREOJCAGA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: write tcp 159.223.143.39:55136->104.36.82.53:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706711345 (2h10m13.487644724s) [seg_pos={0 0}, upload=1979]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:01:57.730740118 (2h10m13.511673491s) [upload=1979]: Done uploading segments... Jul 27 00:01:57.730770763 (2h10m13.511704142s) [upload=1979]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.028558313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 1.847661446s FAIL (2025-07-27 00:01:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== piecestore: piecestore close: conn closed: op Jul 26 23:59:16.227057989 (2h7m32.007991372s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:59:16.254547576 (2h7m32.035480941s) [upload=1948]: Done uploading segments... Jul 26 23:59:16.254573635 (2h7m32.035507007s) [upload=1948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:59:54.547988211 (2h8m10.328921586s) [upload=1964]: Starting upload Jul 26 23:59:54.548042979 (2h8m10.32897636s) [upload=1964]: Done starting upload Jul 26 23:59:54.548276943 (2h8m10.32921031s) [upload=1964]: Uploading segments... Jul 26 23:59:54.548492595 (2h8m10.32942596s) [upload=1964]: Got next segment. Inline: false Jul 26 23:59:54.548548598 (2h8m10.329481973s) [upload=1964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:54.548554144 (2h8m10.32948752s) [upload=1964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:54.579159413 (2h8m10.360092796s) [seg_pos={0 0}, upload=1964]: Begin upload segment... Jul 26 23:59:54.579239063 (2h8m10.360172445s) [seg_pos={0 0}, upload=1964]: Done begin upload segment. Jul 26 23:59:54.579255070 (2h8m10.360188454s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579272031 (2h8m10.360205417s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579795267 (2h8m10.360728646s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580249169 (2h8m10.361182554s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580720799 (2h8m10.361654184s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581340224 (2h8m10.362273617s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581818551 (2h8m10.362751934s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.583612154 (2h8m10.364545544s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584156149 (2h8m10.365089533s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584606999 (2h8m10.365540385s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585024800 (2h8m10.365958185s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585458096 (2h8m10.366391593s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585874293 (2h8m10.366807678s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586299408 (2h8m10.367232794s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586411303 (2h8m10.367344684s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586790559 (2h8m10.367723952s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.659938996 (2h8m10.440872361s) [upload=1964]: Next returned nil segment Jul 26 23:59:54.659966544 (2h8m10.44089992s) [upload=1964]: Waiting for error group managing segments... Jul 26 23:59:54.663545395 (2h8m10.444478762s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.663865280 (2h8m10.444798649s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664358590 (2h8m10.445291955s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664460777 (2h8m10.445394142s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664835941 (2h8m10.445769306s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665276979 (2h8m10.446210344s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665301700 (2h8m10.446235061s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665760740 (2h8m10.446694105s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665849500 (2h8m10.446782862s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:14.358891658 (2h8m30.13982504s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:24.507596093 (2h8m40.288529472s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:60098->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507722906 (2h8m40.288656283s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:58906->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507782037 (2h8m40.288715423s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:46428->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507833692 (2h8m40.288767069s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46586->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507864769 (2h8m40.288798149s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:48846->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507941203 (2h8m40.288874584s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:55006->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507976646 (2h8m40.288910036s) [seg_pos={0 0}, upload=1964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:24.544497933 (2h8m40.325431318s) [upload=1964]: Done uploading segments... Jul 27 00:00:24.544530005 (2h8m40.32546339s) [upload=1964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.847685926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...] duration: 2.016496498s FAIL (2025-07-27 00:01:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.016613027s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.......XXX...XX.....X.XXX...............X..X......X.XX...] duration: 1.875026191s FAIL (2025-07-27 00:01:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.875134387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026225422s FAIL (2025-07-27 00:00:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== =1965]: Done begin upload segment. Jul 26 23:59:57.682728020 (2h8m13.463661405s) [noise=true , address=64.49.13.154:4927 , piece_id=XIRNQBWXUA3O5NBD374HAFGZPAYOCAE2HUF3MXDF5V5BP4LYD4RA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.682749032 (2h8m13.463682422s) [noise=true , address=79.127.205.245:4243 , piece_id=SVH2MBRURZT53RH45EN7EPMXNA3UCXPHJVNAI4QY5HPKL2ASWSQQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.683326904 (2h8m13.464260301s) [noise=true , address=79.127.205.235:4687 , piece_id=5H2ELRDXBVO4HQKOBDXC5MSITWDJKJM4GMTD57FYXFJRA2CKT3QQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.683814966 (2h8m13.46474835s) [noise=true , address=104.36.82.21:4500 , piece_id=PRXGFOHEAEOU7YAPBNKOF7M74ARS6REKXHJ6S5AMQSTORSCX747Q, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684351431 (2h8m13.465284819s) [noise=true , address=77.90.31.18:4831 , piece_id=SPD3AOEASNV4XZSPHACV53645TNO7W2QOUELUACCTMGAO5LM52PA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684460147 (2h8m13.465393532s) [noise=true , address=77.90.31.13:4656 , piece_id=3UB6YMY2DE7BBWK636GNIYG5DKHETZJAP26D5UWLT7MYF6GZXKLQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684801709 (2h8m13.465735102s) [noise=true , address=104.36.82.13:4198 , piece_id=GKO4JN75T4TUE3C6RX6564Z6NQILSNBPBDX7TTT6YW75KRHNPJEA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684949832 (2h8m13.465883209s) [noise=true , address=104.36.82.55:4211 , piece_id=AM5IN34NB3UFNL3CBOSQWQBRA3XRSTKMAOL2FHCDWBPSIRVC3O6Q, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.685187152 (2h8m13.466120534s) [noise=true , address=104.36.82.47:4535 , piece_id=EN6YEX6VNYNYBWYTFYU7NHWQ2XC4F75PPVEXR5URIB7B5YT6RW5A, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.685568763 (2h8m13.46650215s) [noise=true , address=45.91.116.10:4093 , piece_id=5CVTV47Q4KCM6ZE5CEQFBZJFQZBINBNKMKWJZWWM6744PRKKMYPQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.685993178 (2h8m13.466926562s) [noise=true , address=64.49.13.138:4483 , piece_id=22VCYZTMBQGJ2MO4SRJJ6IIUEGKC73VVXDORASUF7CCAHSKF4LFA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686099735 (2h8m13.46703312s) [noise=true , address=79.127.205.234:4589 , piece_id=5SVRERNFFOA3MSE7BNMNNDOXCQRAWEMTQLTUVEBLGXRYZESVOWOQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686400363 (2h8m13.467333749s) [noise=true , address=104.36.82.54:4175 , piece_id=N2STLZTEMBON7WTGVBDRYVXP6TXIGJRLJBRRT5YOA6RKNQKNOOKA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686802147 (2h8m13.467735533s) [noise=true , address=104.36.82.56:4239 , piece_id=G26PSKPF7PCVDRDPIBCVYGGYEADUCCVUXZBEMBQDVVOFXFT2KYYA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686848171 (2h8m13.467781535s) [noise=true , address=79.127.205.246:4277 , piece_id=IJBJWKU4U75FLYD2RS7KHRWXU4LDKPZFAD25V4H7TGU4JJFZJ4ZQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.687242696 (2h8m13.468176095s) [noise=true , address=104.36.82.46:4514 , piece_id=7Q546ZFZD6KIMJ3FZWPO2NHPJJFSWDKXEMDOTIHQKO2AZ5AJP4RA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.687655641 (2h8m13.468589028s) [noise=true , address=104.36.82.53:4124 , piece_id=OGO5CN52D4AO7I3URHXZLU63KJJUOIGK7XZRF2FRYOCVJVOCFHXQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.688053538 (2h8m13.468986956s) [noise=true , address=77.90.31.11:4587 , piece_id=XGF4KLYLGSTI5PCUL56BHSYGJD2UZA2UHWTDJ7YXKDOICIIAPYHA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.760433130 (2h8m13.541366511s) [upload=1965]: Next returned nil segment Jul 26 23:59:57.760457234 (2h8m13.541390622s) [upload=1965]: Waiting for error group managing segments... Jul 26 23:59:57.768399847 (2h8m13.549333211s) [noise=true , address=45.91.116.10:4093 , piece_id=5CVTV47Q4KCM6ZE5CEQFBZJFQZBINBNKMKWJZWWM6744PRKKMYPQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:57.797594051 (2h8m13.578527418s) [noise=true , address=77.90.31.13:4656 , piece_id=3UB6YMY2DE7BBWK636GNIYG5DKHETZJAP26D5UWLT7MYF6GZXKLQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:57.798319183 (2h8m13.579252577s) [noise=true , address=77.90.31.11:4587 , piece_id=XGF4KLYLGSTI5PCUL56BHSYGJD2UZA2UHWTDJ7YXKDOICIIAPYHA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:57.806076236 (2h8m13.58700961s) [noise=true , address=77.90.31.18:4831 , piece_id=SPD3AOEASNV4XZSPHACV53645TNO7W2QOUELUACCTMGAO5LM52PA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:58.258503049 (2h8m14.039436431s) [noise=true , address=79.127.205.246:4277 , piece_id=IJBJWKU4U75FLYD2RS7KHRWXU4LDKPZFAD25V4H7TGU4JJFZJ4ZQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:58.271072750 (2h8m14.05200612s) [noise=true , address=79.127.205.245:4243 , piece_id=SVH2MBRURZT53RH45EN7EPMXNA3UCXPHJVNAI4QY5HPKL2ASWSQQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:58.281779600 (2h8m14.062712975s) [noise=true , address=79.127.205.234:4589 , piece_id=5SVRERNFFOA3MSE7BNMNNDOXCQRAWEMTQLTUVEBLGXRYZESVOWOQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:59.596826170 (2h8m15.377759536s) [noise=true , address=79.127.205.235:4687 , piece_id=5H2ELRDXBVO4HQKOBDXC5MSITWDJKJM4GMTD57FYXFJRA2CKT3QQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 27 00:00:00.657075152 (2h8m16.438008536s) [noise=true , address=64.49.13.138:4483 , piece_id=22VCYZTMBQGJ2MO4SRJJ6IIUEGKC73VVXDORASUF7CCAHSKF4LFA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:03.005366902 (2h8m18.786300285s) [noise=true , address=64.49.13.154:4927 , piece_id=XIRNQBWXUA3O5NBD374HAFGZPAYOCAE2HUF3MXDF5V5BP4LYD4RA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 27 00:00:57.610259036 (2h9m13.391192402s) [noise=true , address=104.36.82.46:4514 , piece_id=7Q546ZFZD6KIMJ3FZWPO2NHPJJFSWDKXEMDOTIHQKO2AZ5AJP4RA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.610931458 (2h9m13.391864824s) [noise=true , address=104.36.82.21:4500 , piece_id=PRXGFOHEAEOU7YAPBNKOF7M74ARS6REKXHJ6S5AMQSTORSCX747Q, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1FPAiRym6MSHqcrk5PMK8sCafNt3sv55TrS848QJmBaUpzDcsx, address:104.36.82.21:4500): protocol: write tcp 159.223.143.39:41580->104.36.82.21:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.610987271 (2h9m13.391920637s) [noise=true , address=104.36.82.53:4124 , piece_id=OGO5CN52D4AO7I3URHXZLU63KJJUOIGK7XZRF2FRYOCVJVOCFHXQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: write tcp 159.223.143.39:46584->104.36.82.53:4124: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.611065051 (2h9m13.391998418s) [noise=true , address=104.36.82.47:4535 , piece_id=EN6YEX6VNYNYBWYTFYU7NHWQ2XC4F75PPVEXR5URIB7B5YT6RW5A, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: write tcp 159.223.143.39:44246->104.36.82.47:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.611115126 (2h9m13.392048491s) [noise=true , address=104.36.82.55:4211 , piece_id=AM5IN34NB3UFNL3CBOSQWQBRA3XRSTKMAOL2FHCDWBPSIRVC3O6Q, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 159.223.143.39:44870->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.611196404 (2h9m13.392129768s) [noise=true , address=104.36.82.54:4175 , piece_id=N2STLZTEMBON7WTGVBDRYVXP6TXIGJRLJBRRT5YOA6RKNQKNOOKA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.611281104 (2h9m13.392214469s) [noise=true , address=104.36.82.13:4198 , piece_id=GKO4JN75T4TUE3C6RX6564Z6NQILSNBPBDX7TTT6YW75KRHNPJEA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm, address:104.36.82.13:4198): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.611371624 (2h9m13.39230499s) [noise=true , address=104.36.82.56:4239 , piece_id=G26PSKPF7PCVDRDPIBCVYGGYEADUCCVUXZBEMBQDVVOFXFT2KYYA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.611388831 (2h9m13.392322209s) [seg_pos={0 0}, upload=1965]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:00:57.634432783 (2h9m13.415366158s) [upload=1965]: Done uploading segments... Jul 27 00:00:57.634458669 (2h9m13.415392045s) [upload=1965]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.026243109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X..X.......X...X....XX..............X.X........X...XX.....] duration: 27.693206352s FAIL (2025-07-27 00:00:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.693224058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.......XXX...XX.....X.XXX...............X..X......X.XX..] duration: 2.218024143s FAIL (2025-07-27 00:00:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.218054912s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X..X.......X...X....XX..............X.X........X...XX....] duration: 54.889232269s FAIL (2025-07-27 00:00:26 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.889326144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040368781s FAIL (2025-07-27 00:00:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== piecestore: piecestore close: conn closed: op Jul 26 23:59:16.227057989 (2h7m32.007991372s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:59:16.254547576 (2h7m32.035480941s) [upload=1948]: Done uploading segments... Jul 26 23:59:16.254573635 (2h7m32.035507007s) [upload=1948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:59:54.547988211 (2h8m10.328921586s) [upload=1964]: Starting upload Jul 26 23:59:54.548042979 (2h8m10.32897636s) [upload=1964]: Done starting upload Jul 26 23:59:54.548276943 (2h8m10.32921031s) [upload=1964]: Uploading segments... Jul 26 23:59:54.548492595 (2h8m10.32942596s) [upload=1964]: Got next segment. Inline: false Jul 26 23:59:54.548548598 (2h8m10.329481973s) [upload=1964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:54.548554144 (2h8m10.32948752s) [upload=1964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:54.579159413 (2h8m10.360092796s) [seg_pos={0 0}, upload=1964]: Begin upload segment... Jul 26 23:59:54.579239063 (2h8m10.360172445s) [seg_pos={0 0}, upload=1964]: Done begin upload segment. Jul 26 23:59:54.579255070 (2h8m10.360188454s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579272031 (2h8m10.360205417s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579795267 (2h8m10.360728646s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580249169 (2h8m10.361182554s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580720799 (2h8m10.361654184s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581340224 (2h8m10.362273617s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581818551 (2h8m10.362751934s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.583612154 (2h8m10.364545544s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584156149 (2h8m10.365089533s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584606999 (2h8m10.365540385s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585024800 (2h8m10.365958185s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585458096 (2h8m10.366391593s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585874293 (2h8m10.366807678s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586299408 (2h8m10.367232794s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586411303 (2h8m10.367344684s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586790559 (2h8m10.367723952s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.659938996 (2h8m10.440872361s) [upload=1964]: Next returned nil segment Jul 26 23:59:54.659966544 (2h8m10.44089992s) [upload=1964]: Waiting for error group managing segments... Jul 26 23:59:54.663545395 (2h8m10.444478762s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.663865280 (2h8m10.444798649s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664358590 (2h8m10.445291955s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664460777 (2h8m10.445394142s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664835941 (2h8m10.445769306s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665276979 (2h8m10.446210344s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665301700 (2h8m10.446235061s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665760740 (2h8m10.446694105s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665849500 (2h8m10.446782862s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:14.358891658 (2h8m30.13982504s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:24.507596093 (2h8m40.288529472s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:60098->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507722906 (2h8m40.288656283s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:58906->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507782037 (2h8m40.288715423s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:46428->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507833692 (2h8m40.288767069s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46586->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507864769 (2h8m40.288798149s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:48846->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507941203 (2h8m40.288874584s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:55006->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507976646 (2h8m40.288910036s) [seg_pos={0 0}, upload=1964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:24.544497933 (2h8m40.325431318s) [upload=1964]: Done uploading segments... Jul 27 00:00:24.544530005 (2h8m40.32546339s) [upload=1964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.0403965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX..] duration: 2.950845546s FAIL (2025-07-27 00:00:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.950880206s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027596837s FAIL (2025-07-26 23:59:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [upload=1950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:58:57.583123083 (2h7m13.364056459s) [seg_pos={0 0}, upload=1950]: Begin upload segment... Jul 26 23:58:57.583170346 (2h7m13.364103712s) [seg_pos={0 0}, upload=1950]: Done begin upload segment. Jul 26 23:58:57.583216747 (2h7m13.364150129s) [noise=true , address=79.127.220.101:4041 , piece_id=F2TOFKHQMXQXXWTD7NRQWURXY4HTT77SYPISQLOQNXEHBMKK43GQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583228177 (2h7m13.364161544s) [noise=true , address=79.127.220.97:4361 , piece_id=LUUTJS4JTBIMZJEK7RSJSZGMJMVKPYRKFQP34FJNM4M45MKFDC5Q, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583592965 (2h7m13.364526343s) [noise=true , address=45.91.116.10:4093 , piece_id=JUIABMNKE2L5PRAV6TIFMOAP4R7XVDWHWJ2RLI77KIQJXUDJFVWQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583700975 (2h7m13.364634341s) [noise=true , address=64.49.13.138:4482 , piece_id=FKOQE2FWUYFHDY7ODSQKP5AVWPY5EFMBCKJHRDJ3IQPDVWZIWGBA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583873420 (2h7m13.364806798s) [noise=true , address=156.146.43.225:4434 , piece_id=COMVSQHZJHENA7VL6N67EY4URHYTCTJ3JQ4SAD2K7WJVWMZSJQMQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584294208 (2h7m13.365227587s) [noise=true , address=104.36.82.54:4171 , piece_id=B75MPJG2NPUENK744IXC6S5AXVTNASUVNCJN3WSY23I6IYGFLMJQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584525691 (2h7m13.365459077s) [noise=true , address=104.36.82.41:4771 , piece_id=2DIYM6YBACQFHHIL6SW3JFIVTBGEUQXLXIEYEJLBOXZTXJPB66EQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584627343 (2h7m13.36556079s) [noise=true , address=77.90.31.11:4577 , piece_id=ZR6A4LCNVRCU7SPCD3MOMCX5TAPEEL6DBPJZ5J4F2DPN57AYIL2Q, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584922118 (2h7m13.365855493s) [noise=true , address=45.91.116.14:4206 , piece_id=JTXNBWRFLANQQFVUPI66BFFRQXU3IOE2YGAPJY7UR473VRFH3PBA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.585187624 (2h7m13.366121002s) [noise=true , address=104.36.82.22:4514 , piece_id=CJHIGGFRC7HSQ2Y36B3YDIK5VEBMDEVUGHUZHK5SMI4HXLZWSC2Q, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.585791738 (2h7m13.366725119s) [noise=true , address=79.127.219.38:4018 , piece_id=AKS3NWDKR627J7G2NPDL3SKVBJGKQSOZORRWE6MXFF4V67UTUNJA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.587457874 (2h7m13.368391255s) [noise=true , address=77.90.31.13:4643 , piece_id=ZYEVYXPWWWYTA7EWXDMACWMNOGZSU7J2XM7CFQV2MMWS3XBMIO5A, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.587824571 (2h7m13.368757946s) [noise=true , address=77.90.31.19:4847 , piece_id=ZTBX2KUWLDR4WUXYBAYBOUGDCSXZU5ZLEAXJPODYCZFTPM4AJVLA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.588296379 (2h7m13.36922976s) [noise=true , address=77.90.31.16:4743 , piece_id=NJQTG5ULV2KGDFKWPEPAIU26REODAL4SQJBSUYCXKBWECQHP2GVA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.588691729 (2h7m13.369625114s) [noise=true , address=104.36.82.47:4535 , piece_id=JW5CLQGCPYMQI5SDOFT7O76TLJEYY5J63EIWEELM2CM7BXLZOOFQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.589038441 (2h7m13.369971817s) [noise=true , address=77.90.31.10:4540 , piece_id=P6LZTNLNMUGVK3YQYS7EUDAAPLRDSFRS3XT6ASBDD4LHH437BZCA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.589360434 (2h7m13.370293809s) [noise=true , address=104.36.82.53:4127 , piece_id=UBIYHMZR5OCUEFUEISHZWY2QLPFQEBLIAEDO2HJ42RF7WFQN52YA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.589664794 (2h7m13.370598163s) [noise=true , address=104.36.82.46:4515 , piece_id=627GFTI4FBA6ETH7ABQ7GWF77LQMVIITEDLAOAIIB4BX2RQT5BEQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.644877694 (2h7m13.425811063s) [upload=1950]: Next returned nil segment Jul 26 23:58:57.644893910 (2h7m13.425827275s) [upload=1950]: Waiting for error group managing segments... Jul 26 23:58:57.648243130 (2h7m13.429176499s) [noise=true , address=79.127.219.38:4018 , piece_id=AKS3NWDKR627J7G2NPDL3SKVBJGKQSOZORRWE6MXFF4V67UTUNJA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:57.686307174 (2h7m13.467240544s) [noise=true , address=77.90.31.13:4643 , piece_id=ZYEVYXPWWWYTA7EWXDMACWMNOGZSU7J2XM7CFQV2MMWS3XBMIO5A, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:57.820828523 (2h7m13.60176189s) [noise=true , address=77.90.31.11:4577 , piece_id=ZR6A4LCNVRCU7SPCD3MOMCX5TAPEEL6DBPJZ5J4F2DPN57AYIL2Q, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.061614895 (2h7m13.842548275s) [noise=true , address=156.146.43.225:4434 , piece_id=COMVSQHZJHENA7VL6N67EY4URHYTCTJ3JQ4SAD2K7WJVWMZSJQMQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.353070332 (2h7m14.134003711s) [noise=true , address=77.90.31.16:4743 , piece_id=NJQTG5ULV2KGDFKWPEPAIU26REODAL4SQJBSUYCXKBWECQHP2GVA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.597956357 (2h7m14.378889728s) [noise=true , address=45.91.116.14:4206 , piece_id=JTXNBWRFLANQQFVUPI66BFFRQXU3IOE2YGAPJY7UR473VRFH3PBA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.870535624 (2h7m14.651468988s) [noise=true , address=77.90.31.19:4847 , piece_id=ZTBX2KUWLDR4WUXYBAYBOUGDCSXZU5ZLEAXJPODYCZFTPM4AJVLA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:59.468270730 (2h7m15.249204099s) [noise=true , address=45.91.116.10:4093 , piece_id=JUIABMNKE2L5PRAV6TIFMOAP4R7XVDWHWJ2RLI77KIQJXUDJFVWQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:00.597144452 (2h7m16.378077826s) [noise=true , address=79.127.220.101:4041 , piece_id=F2TOFKHQMXQXXWTD7NRQWURXY4HTT77SYPISQLOQNXEHBMKK43GQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:00.810801926 (2h7m16.591735293s) [noise=true , address=79.127.220.97:4361 , piece_id=LUUTJS4JTBIMZJEK7RSJSZGMJMVKPYRKFQP34FJNM4M45MKFDC5Q, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:03.813847724 (2h7m19.594781091s) [noise=true , address=64.49.13.138:4482 , piece_id=FKOQE2FWUYFHDY7ODSQKP5AVWPY5EFMBCKJHRDJ3IQPDVWZIWGBA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:57.515591091 (2h8m13.296524466s) [noise=true , address=104.36.82.54:4171 , piece_id=B75MPJG2NPUENK744IXC6S5AXVTNASUVNCJN3WSY23I6IYGFLMJQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515632924 (2h8m13.296566302s) [noise=true , address=104.36.82.47:4535 , piece_id=JW5CLQGCPYMQI5SDOFT7O76TLJEYY5J63EIWEELM2CM7BXLZOOFQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515746678 (2h8m13.296680044s) [noise=true , address=104.36.82.53:4127 , piece_id=UBIYHMZR5OCUEFUEISHZWY2QLPFQEBLIAEDO2HJ42RF7WFQN52YA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: rpc: dial tcp 104.36.82.53:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515781388 (2h8m13.296714755s) [noise=true , address=104.36.82.41:4771 , piece_id=2DIYM6YBACQFHHIL6SW3JFIVTBGEUQXLXIEYEJLBOXZTXJPB66EQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto, address:104.36.82.41:4771): protocol: rpc: dial tcp 104.36.82.41:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515825081 (2h8m13.296758459s) [noise=true , address=104.36.82.22:4514 , piece_id=CJHIGGFRC7HSQ2Y36B3YDIK5VEBMDEVUGHUZHK5SMI4HXLZWSC2Q, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515849512 (2h8m13.296782878s) [noise=true , address=104.36.82.46:4515 , piece_id=627GFTI4FBA6ETH7ABQ7GWF77LQMVIITEDLAOAIIB4BX2RQT5BEQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:59944->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515909624 (2h8m13.296843002s) [noise=true , address=77.90.31.10:4540 , piece_id=P6LZTNLNMUGVK3YQYS7EUDAAPLRDSFRS3XT6ASBDD4LHH437BZCA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:36912->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515945737 (2h8m13.296879121s) [seg_pos={0 0}, upload=1950]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:59:57.540351536 (2h8m13.321284914s) [upload=1950]: Done uploading segments... Jul 26 23:59:57.540379449 (2h8m13.321312825s) [upload=1950]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.027629046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X.......XXX...XX.....X.XXX...............X..X......X.XX.] duration: 2.069330336s FAIL (2025-07-26 23:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.069365405s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X.X.......XXX...XX.....X.XXX...............X..X......X.XX] duration: 30.00108502s FAIL (2025-07-26 23:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00111167s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.030408772s FAIL (2025-07-26 23:59:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641666399 (2h6m36.422599768s) [noise=true , address=104.36.82.16:4293 , piece_id=D2F6MCHAFFAYF2TFXGS3Q6VLQPDLK63Y2B5YPDLSSFII2BV3N3PA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:58:20.641682855 (2h6m36.422616295s) [seg_pos={0 0}, upload=1933]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:58:20.669064353 (2h6m36.449997721s) [upload=1933]: Done uploading segments... Jul 26 23:58:20.669089437 (2h6m36.450022813s) [upload=1933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:58:46.255943658 (2h7m2.036877023s) [upload=1948]: Starting upload Jul 26 23:58:46.256038553 (2h7m2.036971929s) [upload=1948]: Done starting upload Jul 26 23:58:46.256394637 (2h7m2.037328002s) [upload=1948]: Uploading segments... Jul 26 23:58:46.256483799 (2h7m2.037417167s) [upload=1948]: Got next segment. Inline: false Jul 26 23:58:46.256544434 (2h7m2.03747781s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:58:46.256549757 (2h7m2.037483134s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:58:46.284545335 (2h7m2.065478719s) [seg_pos={0 0}, upload=1948]: Begin upload segment... Jul 26 23:58:46.284616775 (2h7m2.065550155s) [seg_pos={0 0}, upload=1948]: Done begin upload segment. Jul 26 23:58:46.284640254 (2h7m2.065573636s) [noise=true , address=45.91.116.6:4973 , piece_id=YQVSXOAHSXNCLA76CWCPZHTWXOVCGURZOMTLPBDCLXN7RPKBHDOQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.284656200 (2h7m2.065589585s) [noise=true , address=77.90.31.17:4770 , piece_id=U2LN2MD3OP5ZMFGRJM2CMLN4UMMTH32QYGU2SDYJIGS3DEJELWBA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.284729873 (2h7m2.065663241s) [noise=true , address=45.91.116.4:4938 , piece_id=GMT26RMQ5XZZQCYIKXOS22F5BWNTAWZZV3HXYOH7DBFPY6FXAR3Q, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285201288 (2h7m2.066134671s) [noise=true , address=45.91.116.6:4974 , piece_id=TXCD6KERV6UMKSOMY76IG63BENK6BZP65JSPG5WXDQRX22QFTHAA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285308763 (2h7m2.066242145s) [noise=true , address=104.36.82.12:4150 , piece_id=M2VVRAEHXNG7I7TUZ3BVHEGEM6UDUURWF524XW2FNBEH5EPWGPYQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285591704 (2h7m2.066525081s) [noise=true , address=45.91.116.7:4011 , piece_id=TEKFCJKCQY3THCEF6EIO53P53MUZ5YGM6OLDZR2LTB4H75GSY3CA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285889973 (2h7m2.06682337s) [noise=true , address=77.90.31.12:4589 , piece_id=C5HNYF4OJ6I5Y45PTVUDPDUVGSHLKLKQAF2GFOQBBJXOKH3MCLMQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.286436892 (2h7m2.067370268s) [noise=true , address=45.91.116.7:4009 , piece_id=TWIWHK2VZ5VK7Z43ZPHZAUELEZSDO7CLJHWHRMVTATQOR4DQZMEA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.298540690 (2h7m2.079474078s) [noise=true , address=104.36.82.2:4732 , piece_id=WOEDYQMH5F3MTAKXT4TAXXJXCUYE6GSO5P2N25DQOP4UUKEIQR4A, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.308351248 (2h7m2.089284634s) [noise=true , address=104.36.82.11:4114 , piece_id=FJ34CYR67HTLNPOS2BNXR6VJTUCDJNSI6SV2LZWMJTWCQ7GA3HSQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.313269976 (2h7m2.094203405s) [noise=true , address=104.36.82.14:4223 , piece_id=54VKHSI5IM33P6LR7LJ5YI5RXS4T7OSQCDV262KT6W5AR5AC6AHA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.318023912 (2h7m2.0989573s) [noise=true , address=77.90.31.15:4699 , piece_id=KWFPLPYY3H6ZVQBADQQJGIUH6ZNG7DMJF3KMRQTAECXYMCCZVTLQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.321982649 (2h7m2.102916036s) [noise=true , address=77.90.31.13:4628 , piece_id=U2FJZCNGDIMHFTOVPOB5AN33TFB4I22BCYWATPURFGMJ4XXTCUMQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.325058306 (2h7m2.105991685s) [noise=true , address=45.91.116.9:4262 , piece_id=KPKD2TZJOHOF4CQIORUZGNE7ZGMEI6D3LGBTC37BNXTPC7HR7M4A, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.328605517 (2h7m2.109538902s) [noise=true , address=77.90.31.14:4663 , piece_id=ZLCDRENFFQOP7PNBX3EAJQUGGBYCFMHZA3FAJVUAU2JUKMSIPZFA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.329095101 (2h7m2.110028469s) [noise=true , address=77.90.31.18:4805 , piece_id=FLNICJ2VW2O7VQIFGPPKJYYPRN3RYC725F4LPJSEE5EVVULUTQVQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.397566505 (2h7m2.178499877s) [upload=1948]: Next returned nil segment Jul 26 23:58:46.397582395 (2h7m2.178515763s) [upload=1948]: Waiting for error group managing segments... Jul 26 23:58:46.415801036 (2h7m2.196734406s) [noise=true , address=77.90.31.14:4663 , piece_id=ZLCDRENFFQOP7PNBX3EAJQUGGBYCFMHZA3FAJVUAU2JUKMSIPZFA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.415841301 (2h7m2.196774664s) [noise=true , address=45.91.116.6:4974 , piece_id=TXCD6KERV6UMKSOMY76IG63BENK6BZP65JSPG5WXDQRX22QFTHAA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.416417702 (2h7m2.197351069s) [noise=true , address=45.91.116.7:4011 , piece_id=TEKFCJKCQY3THCEF6EIO53P53MUZ5YGM6OLDZR2LTB4H75GSY3CA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.418202539 (2h7m2.199135922s) [noise=true , address=77.90.31.12:4589 , piece_id=C5HNYF4OJ6I5Y45PTVUDPDUVGSHLKLKQAF2GFOQBBJXOKH3MCLMQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.418683787 (2h7m2.199617156s) [noise=true , address=45.91.116.7:4009 , piece_id=TWIWHK2VZ5VK7Z43ZPHZAUELEZSDO7CLJHWHRMVTATQOR4DQZMEA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.427339346 (2h7m2.208272725s) [noise=true , address=77.90.31.15:4699 , piece_id=KWFPLPYY3H6ZVQBADQQJGIUH6ZNG7DMJF3KMRQTAECXYMCCZVTLQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.442872874 (2h7m2.22380625s) [noise=true , address=45.91.116.9:4262 , piece_id=KPKD2TZJOHOF4CQIORUZGNE7ZGMEI6D3LGBTC37BNXTPC7HR7M4A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.578499304 (2h7m2.359432672s) [noise=true , address=45.91.116.6:4973 , piece_id=YQVSXOAHSXNCLA76CWCPZHTWXOVCGURZOMTLPBDCLXN7RPKBHDOQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.624214120 (2h7m2.405147492s) [noise=true , address=45.91.116.4:4938 , piece_id=GMT26RMQ5XZZQCYIKXOS22F5BWNTAWZZV3HXYOH7DBFPY6FXAR3Q, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.637977699 (2h7m2.418911076s) [noise=true , address=77.90.31.18:4805 , piece_id=FLNICJ2VW2O7VQIFGPPKJYYPRN3RYC725F4LPJSEE5EVVULUTQVQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.841315447 (2h7m2.622248825s) [noise=true , address=77.90.31.13:4628 , piece_id=U2FJZCNGDIMHFTOVPOB5AN33TFB4I22BCYWATPURFGMJ4XXTCUMQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:59:16.226795168 (2h7m32.007728556s) [noise=true , address=104.36.82.11:4114 , piece_id=FJ34CYR67HTLNPOS2BNXR6VJTUCDJNSI6SV2LZWMJTWCQ7GA3HSQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226882604 (2h7m32.007815988s) [noise=true , address=104.36.82.14:4223 , piece_id=54VKHSI5IM33P6LR7LJ5YI5RXS4T7OSQCDV262KT6W5AR5AC6AHA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226921080 (2h7m32.007854442s) [noise=true , address=77.90.31.17:4770 , piece_id=U2LN2MD3OP5ZMFGRJM2CMLN4UMMTH32QYGU2SDYJIGS3DEJELWBA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226973654 (2h7m32.007907037s) [noise=true , address=104.36.82.2:4732 , piece_id=WOEDYQMH5F3MTAKXT4TAXXJXCUYE6GSO5P2N25DQOP4UUKEIQR4A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:36558->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226988830 (2h7m32.007922215s) [noise=true , address=104.36.82.12:4150 , piece_id=M2VVRAEHXNG7I7TUZ3BVHEGEM6UDUURWF524XW2FNBEH5EPWGPYQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:47142->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.227057989 (2h7m32.007991372s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:59:16.254547576 (2h7m32.035480941s) [upload=1948]: Done uploading segments... Jul 26 23:59:16.254573635 (2h7m32.035507007s) [upload=1948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030434838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX.] duration: 525.713923ms FAIL (2025-07-26 23:59:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 525.765731ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X...X..X.......X...X....XX..............X.X........X...XX...] duration: 40.256455503s FAIL (2025-07-26 23:59:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.256480071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X.X.......XXX...XX.....X.XXX...............X..X......X.X] duration: 30.002146528s FAIL (2025-07-26 23:58:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002172781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX] duration: 30.050569176s FAIL (2025-07-26 23:58:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.05059587s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.036569758s FAIL (2025-07-26 23:58:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== begin upload segment. Jul 26 23:57:51.957513831 (2h6m7.738447215s) [noise=true , address=104.36.82.52:4102 , piece_id=3RMUU2DRTI7IGVSPNBGYSNBDGFO5DK5GOMU25JTSWMPU6PURZG4Q, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.957545107 (2h6m7.738478493s) [noise=true , address=104.36.82.55:4225 , piece_id=DZWY5PIEPQHYRRLNYB5GHYTWPCSFHKCK2R5EDNWWUWSNBYUNOSWA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.957924627 (2h6m7.738858005s) [noise=true , address=79.127.205.235:4679 , piece_id=GO3AKU6756JZJVJOYZ63HCUIRLAJ7WGDW4UBMO7GI7R4F7H3WYPQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958013352 (2h6m7.738946734s) [noise=true , address=77.90.31.16:4746 , piece_id=SSN2EODN36LBAOVGSJYE2ITWKFJEKLJHL4RCCM5XKEQBSAWKD6MQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958277241 (2h6m7.739210619s) [noise=true , address=77.90.31.13:4644 , piece_id=CW6IFOE2IA6AYIKOX6PCA7JTUFHTB4GFHFZADMASAEGE7BHE54LA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958415549 (2h6m7.739348933s) [noise=true , address=104.36.82.38:4700 , piece_id=UOQG7EW6VWSVCVT2BJRWF4KDL6FLMPNZB5QUGBXQP73BHYWZLM2A, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958568217 (2h6m7.739501594s) [noise=true , address=104.36.82.53:4141 , piece_id=2DRWXXQIRVWGAGYKTQ7H27WASIKEOD7XRRVPVI7BBGKZRKM4FV6Q, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958946407 (2h6m7.739879788s) [noise=true , address=77.90.31.11:4578 , piece_id=4EOHNPI2SFNZD3OKWAG6YCAX4NATOP5AIIBOLIATDFYR5LAXDDMA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.958966345 (2h6m7.739899733s) [noise=true , address=64.49.13.140:4508 , piece_id=SSNJICPKVEUDZYMOY5AYLTI7ITU6IXZ3OUAKY7FMGV753EPZTK3Q, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959349291 (2h6m7.740282667s) [noise=true , address=104.36.82.54:4177 , piece_id=7VJMYUHPL52XQV7AMLP2TGKGNJDLOPA5E3PRWCCAD6I6DPY43UPQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959371783 (2h6m7.740305168s) [noise=true , address=104.36.82.56:4237 , piece_id=G2V3S5LU4R5J573RG6FA3V6HTHUNO4D2EP6FVKF2LQIMADS3N6ZQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959671337 (2h6m7.740604715s) [noise=true , address=77.90.31.15:4703 , piece_id=W6OWZDJBWXXFVUE6FUP6RYLTQABXOZAKYL335XOQQ2SJFIVFVQCA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959680516 (2h6m7.740613897s) [noise=true , address=104.36.82.46:4510 , piece_id=BZ73KMXSX54TGAIC74JYOBAPMCFD732CNQ2F4BHPUKOWOEQVFG7A, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.959937415 (2h6m7.740870791s) [noise=true , address=79.127.219.38:4010 , piece_id=WC566RDAZR2JPSLDOQ2JKFJ7TCCHXTZB26BVPMI6VMXMJNMG65BA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.960255168 (2h6m7.741188554s) [noise=true , address=104.36.82.47:4532 , piece_id=N5II752EGQ6IDVNF6VPETZGNJODGRM4425KBCDM47YKTR2XUDDGA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.960346598 (2h6m7.741279967s) [noise=true , address=79.127.201.214:4321 , piece_id=W4NHV25OXI3WQEV3JB6VOEZBPULGUSHRLYTGELLVK7UWQJW6IGTQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.960730290 (2h6m7.741663655s) [noise=true , address=64.49.13.137:4502 , piece_id=O3OMBCB6YSEWDRRASM7JGOLHFTOUTKLGJ5JLX2EWUQP472HIOSRA, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:51.961176005 (2h6m7.742109383s) [noise=true , address=79.127.205.246:4279 , piece_id=K2WS2TP4WHV7VMHZS4CJMBRGO2PMCDUXQ6WJXQBGEFONDAPU2YGQ, seg_pos={0 0}, upload=1935]: Uploading piece... Jul 26 23:57:52.040596586 (2h6m7.821529955s) [upload=1935]: Next returned nil segment Jul 26 23:57:52.040614900 (2h6m7.821548282s) [upload=1935]: Waiting for error group managing segments... Jul 26 23:57:52.079489604 (2h6m7.860422972s) [noise=true , address=77.90.31.16:4746 , piece_id=SSN2EODN36LBAOVGSJYE2ITWKFJEKLJHL4RCCM5XKEQBSAWKD6MQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.079882730 (2h6m7.860816109s) [noise=true , address=77.90.31.11:4578 , piece_id=4EOHNPI2SFNZD3OKWAG6YCAX4NATOP5AIIBOLIATDFYR5LAXDDMA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.417872880 (2h6m8.198806256s) [noise=true , address=79.127.219.38:4010 , piece_id=WC566RDAZR2JPSLDOQ2JKFJ7TCCHXTZB26BVPMI6VMXMJNMG65BA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.531510614 (2h6m8.312443982s) [noise=true , address=79.127.205.235:4679 , piece_id=GO3AKU6756JZJVJOYZ63HCUIRLAJ7WGDW4UBMO7GI7R4F7H3WYPQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.661261963 (2h6m8.442195343s) [noise=true , address=79.127.205.246:4279 , piece_id=K2WS2TP4WHV7VMHZS4CJMBRGO2PMCDUXQ6WJXQBGEFONDAPU2YGQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:52.887288970 (2h6m8.66822235s) [noise=true , address=64.49.13.137:4502 , piece_id=O3OMBCB6YSEWDRRASM7JGOLHFTOUTKLGJ5JLX2EWUQP472HIOSRA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:54.986003119 (2h6m10.766936496s) [noise=true , address=77.90.31.13:4644 , piece_id=CW6IFOE2IA6AYIKOX6PCA7JTUFHTB4GFHFZADMASAEGE7BHE54LA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:55.307965874 (2h6m11.088899252s) [noise=true , address=79.127.201.214:4321 , piece_id=W4NHV25OXI3WQEV3JB6VOEZBPULGUSHRLYTGELLVK7UWQJW6IGTQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:57:57.817915300 (2h6m13.598848678s) [noise=true , address=64.49.13.140:4508 , piece_id=SSNJICPKVEUDZYMOY5AYLTI7ITU6IXZ3OUAKY7FMGV753EPZTK3Q, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:58:19.402176971 (2h6m35.183110337s) [noise=true , address=77.90.31.15:4703 , piece_id=W6OWZDJBWXXFVUE6FUP6RYLTQABXOZAKYL335XOQQ2SJFIVFVQCA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: <nil> Jul 26 23:58:51.894658270 (2h7m7.675591646s) [noise=true , address=104.36.82.52:4102 , piece_id=3RMUU2DRTI7IGVSPNBGYSNBDGFO5DK5GOMU25JTSWMPU6PURZG4Q, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: write tcp 159.223.143.39:35084->104.36.82.52:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.894738332 (2h7m7.675671707s) [noise=true , address=104.36.82.47:4532 , piece_id=N5II752EGQ6IDVNF6VPETZGNJODGRM4425KBCDM47YKTR2XUDDGA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.894809469 (2h7m7.675742844s) [noise=true , address=104.36.82.55:4225 , piece_id=DZWY5PIEPQHYRRLNYB5GHYTWPCSFHKCK2R5EDNWWUWSNBYUNOSWA, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:58:51.894949495 (2h7m7.675882871s) [noise=true , address=104.36.82.38:4700 , piece_id=UOQG7EW6VWSVCVT2BJRWF4KDL6FLMPNZB5QUGBXQP73BHYWZLM2A, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:12JmsGbYEHsP6Pf2HinsbdsLZW85jnaHe8gHZXHcShCTxMxXJKP, address:104.36.82.38:4700): protocol: rpc: dial tcp 104.36.82.38:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895012990 (2h7m7.675946365s) [noise=true , address=104.36.82.56:4237 , piece_id=G2V3S5LU4R5J573RG6FA3V6HTHUNO4D2EP6FVKF2LQIMADS3N6ZQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: write tcp 159.223.143.39:55096->104.36.82.56:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895065796 (2h7m7.67599917s) [noise=true , address=104.36.82.54:4177 , piece_id=7VJMYUHPL52XQV7AMLP2TGKGNJDLOPA5E3PRWCCAD6I6DPY43UPQ, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 159.223.143.39:40814->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895102232 (2h7m7.676035607s) [noise=true , address=104.36.82.46:4510 , piece_id=BZ73KMXSX54TGAIC74JYOBAPMCFD732CNQ2F4BHPUKOWOEQVFG7A, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:35032->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895141702 (2h7m7.67607508s) [noise=true , address=104.36.82.53:4141 , piece_id=2DRWXXQIRVWGAGYKTQ7H27WASIKEOD7XRRVPVI7BBGKZRKM4FV6Q, seg_pos={0 0}, upload=1935]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: write tcp 159.223.143.39:44090->104.36.82.53:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:51.895170857 (2h7m7.676104242s) [seg_pos={0 0}, upload=1935]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:58:51.928281324 (2h7m7.709214702s) [upload=1935]: Done uploading segments... Jul 26 23:58:51.928312614 (2h7m7.709245991s) [upload=1935]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.036613145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.029916228s FAIL (2025-07-26 23:58:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== : write tcp 159.223.143.39:36770->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538990054 (2h6m6.319923431s) [seg_pos={0 0}, upload=1927]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:57:50.564064323 (2h6m6.344997692s) [upload=1927]: Done uploading segments... Jul 26 23:57:50.564195664 (2h6m6.345129035s) [upload=1927]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:57:50.669282712 (2h6m6.450216098s) [upload=1933]: Starting upload Jul 26 23:57:50.669376244 (2h6m6.450309628s) [upload=1933]: Done starting upload Jul 26 23:57:50.669594624 (2h6m6.450527991s) [upload=1933]: Uploading segments... Jul 26 23:57:50.669799987 (2h6m6.450733366s) [upload=1933]: Got next segment. Inline: false Jul 26 23:57:50.669871688 (2h6m6.450805071s) [upload=1933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:50.669877758 (2h6m6.450811137s) [upload=1933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:50.699301330 (2h6m6.480234711s) [seg_pos={0 0}, upload=1933]: Begin upload segment... Jul 26 23:57:50.699350004 (2h6m6.48028337s) [seg_pos={0 0}, upload=1933]: Done begin upload segment. Jul 26 23:57:50.699375683 (2h6m6.480309061s) [noise=true , address=45.91.116.13:4155 , piece_id=ZDC4R62AE7ONI56MW3SKTSXEE2XHI3DZ54ZTEED4IATQ3YRYBPNQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.699414688 (2h6m6.480348073s) [noise=true , address=77.90.31.18:4805 , piece_id=OU5XZPBHVNCWSY3FEH5OAYETFDPZQADQGD6OJC2G7QZZEJJEDW4Q, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.699761291 (2h6m6.480694669s) [noise=true , address=104.36.82.2:4732 , piece_id=JIPLHZ2S6XEHNR35KRHAPTX6DNLK7TFWKI4X5L5FNQUELJY3N3FQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.699884427 (2h6m6.480817947s) [noise=true , address=45.91.116.4:4938 , piece_id=UQCFID7ANH3C62D6JWY6SXDRNO5DHNI72CNTZ223KYJ643DHLRSQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700300544 (2h6m6.481233925s) [noise=true , address=104.36.82.11:4113 , piece_id=LW2LDEUEPF4RM6JQZMJBXAATTUNQQPN42HH4YDJNR3AE5BYZFE4Q, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700453668 (2h6m6.481387053s) [noise=true , address=77.90.31.14:4661 , piece_id=XXOGW6JROE4IQSE2B6IZAVN3D6J3IW4HAHWOSUFTBBQ3QB6XJUBQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700693612 (2h6m6.481626997s) [noise=true , address=45.91.116.14:4190 , piece_id=FYVCED2JOP452LAH74LD2G2X4QYK4BQKBQ6E46W5FBWCLIYOXYGQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.700866568 (2h6m6.481799946s) [noise=true , address=104.36.82.8:4969 , piece_id=62UXYO4GV3TZFAFNR6XGFQJJ67PI4MRS76GQNHQRZVUCE4UVPFRA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701036720 (2h6m6.481970103s) [noise=true , address=104.36.82.16:4293 , piece_id=D2F6MCHAFFAYF2TFXGS3Q6VLQPDLK63Y2B5YPDLSSFII2BV3N3PA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701277262 (2h6m6.482210643s) [noise=true , address=45.91.116.10:4083 , piece_id=4F77NBEDGF4CH45OI3DU5F32NNFNJFJFIJTG6MSM7PMOMASCID2A, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701394578 (2h6m6.482327957s) [noise=true , address=45.91.116.6:4974 , piece_id=H4FLJYQAOKMYZPDUHEHU55AOWUKK7MSQVSY5OSVKYF72KWQCEXXA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.701719862 (2h6m6.48265326s) [noise=true , address=77.90.31.15:4698 , piece_id=T4YSEM442SNZIQOUPQ2A4XMAJ5AQK6O3A5ID5UNDI6ZENRRDPF7A, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.702107535 (2h6m6.483040922s) [noise=true , address=77.90.31.15:4699 , piece_id=UWIGVBJFTBTX7LOAUMZFTVG7ZNNIWC5X62BNRKTUY6337QZF4YGA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.702489223 (2h6m6.48342261s) [noise=true , address=104.36.82.13:4187 , piece_id=SMJMVDB7S47W3IDN2SU7CI3OLTNUZAGKRSVNPVFBIBZHZHWEPJIQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.702861497 (2h6m6.483794878s) [noise=true , address=104.36.82.12:4150 , piece_id=J3XITNETGNAB7LPL5PPNY2VFJPWAQDMTDNN4DWXPZSN2LR6IJ3LA, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.703183106 (2h6m6.484116484s) [noise=true , address=104.36.82.4:4802 , piece_id=FJBX5ZZNJDPTWKXS6TJDEDFEEIL7HJNRDXHRREKWGWASDYAHAOGQ, seg_pos={0 0}, upload=1933]: Uploading piece... Jul 26 23:57:50.779050485 (2h6m6.559983854s) [upload=1933]: Next returned nil segment Jul 26 23:57:50.779066732 (2h6m6.560000107s) [upload=1933]: Waiting for error group managing segments... Jul 26 23:57:50.783242486 (2h6m6.564175854s) [noise=true , address=45.91.116.13:4155 , piece_id=ZDC4R62AE7ONI56MW3SKTSXEE2XHI3DZ54ZTEED4IATQ3YRYBPNQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.783857133 (2h6m6.564790501s) [noise=true , address=45.91.116.6:4974 , piece_id=H4FLJYQAOKMYZPDUHEHU55AOWUKK7MSQVSY5OSVKYF72KWQCEXXA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.784490796 (2h6m6.565424173s) [noise=true , address=45.91.116.4:4938 , piece_id=UQCFID7ANH3C62D6JWY6SXDRNO5DHNI72CNTZ223KYJ643DHLRSQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.784907248 (2h6m6.565840613s) [noise=true , address=77.90.31.14:4661 , piece_id=XXOGW6JROE4IQSE2B6IZAVN3D6J3IW4HAHWOSUFTBBQ3QB6XJUBQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.785327870 (2h6m6.566261232s) [noise=true , address=45.91.116.10:4083 , piece_id=4F77NBEDGF4CH45OI3DU5F32NNFNJFJFIJTG6MSM7PMOMASCID2A, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:50.933440283 (2h6m6.714373652s) [noise=true , address=45.91.116.14:4190 , piece_id=FYVCED2JOP452LAH74LD2G2X4QYK4BQKBQ6E46W5FBWCLIYOXYGQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:52.178035664 (2h6m7.958969033s) [noise=true , address=77.90.31.15:4698 , piece_id=T4YSEM442SNZIQOUPQ2A4XMAJ5AQK6O3A5ID5UNDI6ZENRRDPF7A, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:57:52.266254515 (2h6m8.047187882s) [noise=true , address=77.90.31.15:4699 , piece_id=UWIGVBJFTBTX7LOAUMZFTVG7ZNNIWC5X62BNRKTUY6337QZF4YGA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:58:08.973792785 (2h6m24.754726157s) [noise=true , address=77.90.31.18:4805 , piece_id=OU5XZPBHVNCWSY3FEH5OAYETFDPZQADQGD6OJC2G7QZZEJJEDW4Q, seg_pos={0 0}, upload=1933]: Done uploading piece. err: <nil> Jul 26 23:58:20.641348043 (2h6m36.422281419s) [noise=true , address=104.36.82.13:4187 , piece_id=SMJMVDB7S47W3IDN2SU7CI3OLTNUZAGKRSVNPVFBIBZHZHWEPJIQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641410814 (2h6m36.42234418s) [noise=true , address=104.36.82.12:4150 , piece_id=J3XITNETGNAB7LPL5PPNY2VFJPWAQDMTDNN4DWXPZSN2LR6IJ3LA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641457323 (2h6m36.4223907s) [noise=true , address=104.36.82.4:4802 , piece_id=FJBX5ZZNJDPTWKXS6TJDEDFEEIL7HJNRDXHRREKWGWASDYAHAOGQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641500686 (2h6m36.422434061s) [noise=true , address=104.36.82.2:4732 , piece_id=JIPLHZ2S6XEHNR35KRHAPTX6DNLK7TFWKI4X5L5FNQUELJY3N3FQ, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641546021 (2h6m36.422479388s) [noise=true , address=104.36.82.11:4113 , piece_id=LW2LDEUEPF4RM6JQZMJBXAATTUNQQPN42HH4YDJNR3AE5BYZFE4Q, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641593381 (2h6m36.42252675s) [noise=true , address=104.36.82.8:4969 , piece_id=62UXYO4GV3TZFAFNR6XGFQJJ67PI4MRS76GQNHQRZVUCE4UVPFRA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641666399 (2h6m36.422599768s) [noise=true , address=104.36.82.16:4293 , piece_id=D2F6MCHAFFAYF2TFXGS3Q6VLQPDLK63Y2B5YPDLSSFII2BV3N3PA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:58:20.641682855 (2h6m36.422616295s) [seg_pos={0 0}, upload=1933]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:58:20.669064353 (2h6m36.449997721s) [upload=1933]: Done uploading segments... Jul 26 23:58:20.669089437 (2h6m36.450022813s) [upload=1933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.029930681s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.X...X..X.......X...X....XX..............X.X........X...XX..] duration: 49.623759715s FAIL (2025-07-26 23:58:17 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.623795754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXX] duration: 30.035568219s FAIL (2025-07-26 23:57:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.035585152s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [X.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.045183397s FAIL (2025-07-26 23:57:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 6OQJ56GL4Z7K7QIN3ZBWX4J4RPG4NDHDLLKUOYRSVY5HISQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.851564451 (2h5m7.632497837s) [noise=true , address=77.90.31.16:4748 , piece_id=QY5S6GF3SMGU7OIZPLSITEH7IV66T2TXAK7VNTNBZRZWWJYAONCA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.851690596 (2h5m7.632623972s) [noise=true , address=104.36.82.47:4521 , piece_id=R25H2RLSDKWE3NJ5YO5WORMUJIEH4CMPQUSCQNWQ57J7CLVUVBVA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852021546 (2h5m7.632954924s) [noise=true , address=64.49.13.138:4482 , piece_id=EXHEP7BWNESA4TPIKVVB7AGSNBSQGCXZZ2HNLLBZGU5WDKLG53QA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852268583 (2h5m7.63320197s) [noise=true , address=77.90.31.18:4821 , piece_id=ANASWX6HVEX4JHDGTLGJJKHQZDSJIFVWJJ4CABNLOFS3FO4BIGMA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852652801 (2h5m7.633586185s) [noise=true , address=104.36.82.55:4199 , piece_id=3FJAV34EDOWTGERP2Z72G3SWIIGW3NSEQKFJ5COTMCEYMM7T423Q, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.852709896 (2h5m7.633643281s) [noise=true , address=104.36.82.54:4160 , piece_id=LDGIIUJJX3W52OASO5LTVAVCYLV2RBXJREO2YRVNRHZV2TVMBJPA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853158467 (2h5m7.634091846s) [noise=true , address=45.91.116.9:4277 , piece_id=IUDVNG2CG6G6QTAUSBRR6RLD5FWTLIJ6NL53DNE56BTLCXMXGR7Q, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853180275 (2h5m7.634113662s) [noise=true , address=104.36.82.46:4505 , piece_id=EFMU4SFBKLFSZVS47FXCVMOX7CBMZCW45UQJSKDEV6RSLVZEP3DA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853635476 (2h5m7.634568858s) [noise=true , address=79.127.205.235:4689 , piece_id=XXGET5NG7EWR3Y27Q6ME64ZLYMQ4YXXB56HVRH4JOPG2WV6LZOKA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.853752338 (2h5m7.634685713s) [noise=true , address=64.49.13.137:4502 , piece_id=HAH2WCABFGOLFPY3OPYVILV6KDWRUPLS52HDTPLXSLGUTNO72SRA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854060924 (2h5m7.634994312s) [noise=true , address=79.127.201.215:4351 , piece_id=T577FJOHUM53ZGPMZWBFAWU7JJSVZLPMWQPU5KJCIC7EVSXHHFZQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854198788 (2h5m7.635132163s) [noise=true , address=79.127.205.233:4646 , piece_id=FSRHXJS2EKIV6FUOOGB6Y5NIP4PX4Z7RYKQ3PF32P7FBU52ZRDAA, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854528885 (2h5m7.635462267s) [noise=true , address=64.49.13.140:4508 , piece_id=6WDQZVIGND7EZ7M7EV4WSBARSROEOUSGQXOVSQZTURJQMENZKPBQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.854974085 (2h5m7.635907472s) [noise=true , address=104.36.82.14:4226 , piece_id=ZKQ3M5Y27UPPL6TKG5TJM76VS44AEHCN5UMFQZCDMDU4R37G35NQ, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.855483525 (2h5m7.636416913s) [noise=true , address=45.91.116.14:4206 , piece_id=G4EX4E4YVNEE4TF5XRALG4MCLQKRWIMTBMOKDQT5TK5HN74CPM6A, seg_pos={0 0}, upload=1920]: Uploading piece... Jul 26 23:56:51.920993713 (2h5m7.701927097s) [noise=true , address=45.91.116.14:4206 , piece_id=G4EX4E4YVNEE4TF5XRALG4MCLQKRWIMTBMOKDQT5TK5HN74CPM6A, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:56:51.930652173 (2h5m7.711585536s) [upload=1920]: Next returned nil segment Jul 26 23:56:51.930673404 (2h5m7.711606789s) [upload=1920]: Waiting for error group managing segments... Jul 26 23:56:51.935563179 (2h5m7.716496555s) [noise=true , address=77.90.31.16:4748 , piece_id=QY5S6GF3SMGU7OIZPLSITEH7IV66T2TXAK7VNTNBZRZWWJYAONCA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:52.455117803 (2h5m8.236051176s) [noise=true , address=79.127.205.233:4646 , piece_id=FSRHXJS2EKIV6FUOOGB6Y5NIP4PX4Z7RYKQ3PF32P7FBU52ZRDAA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:52.550219414 (2h5m8.331152789s) [noise=true , address=64.49.13.137:4502 , piece_id=HAH2WCABFGOLFPY3OPYVILV6KDWRUPLS52HDTPLXSLGUTNO72SRA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 26 23:56:52.856645765 (2h5m8.637579131s) [noise=true , address=79.127.205.235:4689 , piece_id=XXGET5NG7EWR3Y27Q6ME64ZLYMQ4YXXB56HVRH4JOPG2WV6LZOKA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:52.864031985 (2h5m8.644965364s) [noise=true , address=79.127.201.215:4351 , piece_id=T577FJOHUM53ZGPMZWBFAWU7JJSVZLPMWQPU5KJCIC7EVSXHHFZQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:56:53.066732432 (2h5m8.847665809s) [noise=true , address=64.49.13.138:4482 , piece_id=EXHEP7BWNESA4TPIKVVB7AGSNBSQGCXZZ2HNLLBZGU5WDKLG53QA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:56:53.252768855 (2h5m9.033702234s) [noise=true , address=64.49.13.140:4508 , piece_id=6WDQZVIGND7EZ7M7EV4WSBARSROEOUSGQXOVSQZTURJQMENZKPBQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:57:05.777632884 (2h5m21.5585664s) [noise=true , address=45.91.116.9:4277 , piece_id=IUDVNG2CG6G6QTAUSBRR6RLD5FWTLIJ6NL53DNE56BTLCXMXGR7Q, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:57:13.378399729 (2h5m29.159333092s) [noise=true , address=77.90.31.19:4875 , piece_id=YUWMH6OQJ56GL4Z7K7QIN3ZBWX4J4RPG4NDHDLLKUOYRSVY5HISQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: <nil> Jul 26 23:57:51.782002881 (2h6m7.562936283s) [noise=true , address=77.90.31.18:4821 , piece_id=ANASWX6HVEX4JHDGTLGJJKHQZDSJIFVWJJ4CABNLOFS3FO4BIGMA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:38968->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782091723 (2h6m7.563025098s) [noise=true , address=104.36.82.47:4521 , piece_id=R25H2RLSDKWE3NJ5YO5WORMUJIEH4CMPQUSCQNWQ57J7CLVUVBVA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: write tcp 159.223.143.39:34394->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782153762 (2h6m7.563087138s) [noise=true , address=104.36.82.54:4160 , piece_id=LDGIIUJJX3W52OASO5LTVAVCYLV2RBXJREO2YRVNRHZV2TVMBJPA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: write tcp 159.223.143.39:43814->104.36.82.54:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782214832 (2h6m7.563148212s) [noise=true , address=104.36.82.14:4226 , piece_id=ZKQ3M5Y27UPPL6TKG5TJM76VS44AEHCN5UMFQZCDMDU4R37G35NQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: write tcp 159.223.143.39:51194->104.36.82.14:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782275917 (2h6m7.563209293s) [noise=true , address=104.36.82.55:4199 , piece_id=3FJAV34EDOWTGERP2Z72G3SWIIGW3NSEQKFJ5COTMCEYMM7T423Q, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 159.223.143.39:44846->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782325578 (2h6m7.563258954s) [noise=true , address=77.90.31.17:4795 , piece_id=4WGI3K74NQS5ZCXCO3UFOI3FQBGHSXIXLGPRLHPLFZT7GMGMZ7HQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782366822 (2h6m7.563300204s) [noise=true , address=104.36.82.46:4505 , piece_id=EFMU4SFBKLFSZVS47FXCVMOX7CBMZCW45UQJSKDEV6RSLVZEP3DA, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: rpc: dial tcp 104.36.82.46:4505: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782421930 (2h6m7.56335531s) [noise=true , address=104.36.82.56:4254 , piece_id=N3IX3634VTYN3D5YLTRT2SLDKN2SQBWO567RDEKAHMLMEGKPYFSQ, seg_pos={0 0}, upload=1920]: Done uploading piece. err: ecclient: upload failed (node:1k4x3HL3PNHwKJX83RyqvnsATHk3HmeMiC12A7raLN58wzvkUN, address:104.36.82.56:4254): protocol: rpc: dial tcp 104.36.82.56:4254: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:51.782483496 (2h6m7.563416938s) [seg_pos={0 0}, upload=1920]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:57:51.824438035 (2h6m7.605371414s) [upload=1920]: Done uploading segments... Jul 26 23:57:51.824462856 (2h6m7.605396225s) [upload=1920]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.045205187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 30.027203231s FAIL (2025-07-26 23:57:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420354433 (2h5m36.201287809s) [noise=true , address=104.36.82.7:4911 , piece_id=CBS6NFPCFI6ENWBH6BEHIQ3UGCV5475GTLOSUZ4GYHQOP4XOS46Q, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420374692 (2h5m36.201308072s) [seg_pos={0 0}, upload=1919]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:57:20.455540937 (2h5m36.236474325s) [upload=1919]: Done uploading segments... Jul 26 23:57:20.455569078 (2h5m36.236502457s) [upload=1919]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:57:20.568454898 (2h5m36.349388274s) [upload=1927]: Starting upload Jul 26 23:57:20.568669185 (2h5m36.349602567s) [upload=1927]: Done starting upload Jul 26 23:57:20.568792233 (2h5m36.349725603s) [upload=1927]: Uploading segments... Jul 26 23:57:20.569146906 (2h5m36.35008029s) [upload=1927]: Got next segment. Inline: false Jul 26 23:57:20.569244299 (2h5m36.350177684s) [upload=1927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:20.569251429 (2h5m36.350184819s) [upload=1927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:20.601609712 (2h5m36.382543099s) [seg_pos={0 0}, upload=1927]: Begin upload segment... Jul 26 23:57:20.601673862 (2h5m36.382607237s) [seg_pos={0 0}, upload=1927]: Done begin upload segment. Jul 26 23:57:20.601715768 (2h5m36.382649134s) [noise=true , address=104.36.82.6:4876 , piece_id=637YSFHNBIV4P5XWRCPH24LPJYY3M6MFFWC4Y5X6Z655ZZPGXJ6Q, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.602124165 (2h5m36.383057548s) [noise=true , address=45.91.116.14:4189 , piece_id=KLE43JQOKTKITXNAJJVJYFMFCJ3SFSMC3EMLBFXK6HC3TCF5MJZA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.602487258 (2h5m36.383420644s) [noise=true , address=45.91.116.13:4153 , piece_id=3GBGT5TMQO3ARTKZVA3SJQ6LNOVZTVUEASYVENEJAQGJEGPRSOCA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.603068947 (2h5m36.384002327s) [noise=true , address=77.90.31.19:4842 , piece_id=PSQZGLVQKUU3DMIOT3VQ62XZAFVLSGFQ66Z7PMBASX6KZPJ522IQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.603200750 (2h5m36.384134127s) [noise=true , address=45.91.116.4:4938 , piece_id=JH3GTJD37UAFNHJEE7POZI2BYESIUV3SLVHEVTYLWJUECKT3MBVA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.603794398 (2h5m36.384727774s) [noise=true , address=77.90.31.14:4662 , piece_id=VWABP3RH3VT2SQA3POY5WTC2TDRR7PXSRSWIYOXKEUG2WRIDBIJA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.604033107 (2h5m36.384966494s) [noise=true , address=104.36.82.5:4837 , piece_id=UH67FWYQTAYDGTP65NA5BO2MSQIOSPGSJYYR6GYP55C7JR5EIGGQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.604891309 (2h5m36.385824686s) [noise=true , address=45.91.116.8:4048 , piece_id=P2UZRXYEXUTUTQMUEUOYKOXCWDUGQXWDPRJ2PXX35DMIYXANKGJQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.605266931 (2h5m36.386200313s) [noise=true , address=77.90.31.13:4626 , piece_id=75CQFMO5CTU7RZDFQJ5U3FN3GGSTQAHYYF4EE4YDRBPBLGRJDVRA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.605643773 (2h5m36.386577221s) [noise=true , address=77.90.31.15:4699 , piece_id=7CZYIG3GWGSDKLXUOG35Y5PHK46MKPUWWWCEH4XOBCRKN2MBW7JA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.606028106 (2h5m36.386961491s) [noise=true , address=45.91.116.11:4117 , piece_id=6MS4BJB4KL3RQL5AOKLZHZ4T5BJR2M4LK4ZZ6H5R4OOAHZ7TSGSA, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.606380314 (2h5m36.387313697s) [noise=true , address=45.91.116.12:4225 , piece_id=ORCWODLYVGYKU6YMJH2W2MPN4DMNFJFB22VY6ZWER7WLXBBVTGLQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.607164117 (2h5m36.388097501s) [noise=true , address=77.90.31.17:4771 , piece_id=VVEBB6CJDW3CYLC6HM2MGCAGDXM7T5ZOHOMQXA6V3WBUI34Q3RQQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.607560464 (2h5m36.388493854s) [noise=true , address=104.36.82.7:4912 , piece_id=GFNR5C5LGCJEOOWZ52RNSU2HHJPN5I3SRAAOIX3WSCYVGHIR5QQQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.607983625 (2h5m36.388917012s) [noise=true , address=77.90.31.12:4589 , piece_id=P3NL3JACSRBQ6BQZDMENVSFLHWEPVI47HRCFHUYI25QXLW2XBTRQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.608581577 (2h5m36.389514953s) [noise=true , address=45.91.116.5:4299 , piece_id=IT3CWTXUVLJCZHH3SE6R6S2SESH6WCPIK436YS26DWPGNNEA2MXQ, seg_pos={0 0}, upload=1927]: Uploading piece... Jul 26 23:57:20.703573074 (2h5m36.484506464s) [upload=1927]: Next returned nil segment Jul 26 23:57:20.703595449 (2h5m36.484528836s) [upload=1927]: Waiting for error group managing segments... Jul 26 23:57:20.707832208 (2h5m36.488765575s) [noise=true , address=45.91.116.5:4299 , piece_id=IT3CWTXUVLJCZHH3SE6R6S2SESH6WCPIK436YS26DWPGNNEA2MXQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.709464304 (2h5m36.490397669s) [noise=true , address=77.90.31.14:4662 , piece_id=VWABP3RH3VT2SQA3POY5WTC2TDRR7PXSRSWIYOXKEUG2WRIDBIJA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.709931221 (2h5m36.490864584s) [noise=true , address=45.91.116.12:4225 , piece_id=ORCWODLYVGYKU6YMJH2W2MPN4DMNFJFB22VY6ZWER7WLXBBVTGLQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.710397031 (2h5m36.491330401s) [noise=true , address=45.91.116.4:4938 , piece_id=JH3GTJD37UAFNHJEE7POZI2BYESIUV3SLVHEVTYLWJUECKT3MBVA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.710932201 (2h5m36.491865571s) [noise=true , address=45.91.116.11:4117 , piece_id=6MS4BJB4KL3RQL5AOKLZHZ4T5BJR2M4LK4ZZ6H5R4OOAHZ7TSGSA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.711462379 (2h5m36.492395744s) [noise=true , address=45.91.116.8:4048 , piece_id=P2UZRXYEXUTUTQMUEUOYKOXCWDUGQXWDPRJ2PXX35DMIYXANKGJQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.711871688 (2h5m36.492805057s) [noise=true , address=45.91.116.13:4153 , piece_id=3GBGT5TMQO3ARTKZVA3SJQ6LNOVZTVUEASYVENEJAQGJEGPRSOCA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.712409304 (2h5m36.493342674s) [noise=true , address=77.90.31.15:4699 , piece_id=7CZYIG3GWGSDKLXUOG35Y5PHK46MKPUWWWCEH4XOBCRKN2MBW7JA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:20.757155063 (2h5m36.538088444s) [noise=true , address=45.91.116.14:4189 , piece_id=KLE43JQOKTKITXNAJJVJYFMFCJ3SFSMC3EMLBFXK6HC3TCF5MJZA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:22.061385227 (2h5m37.842318606s) [noise=true , address=77.90.31.12:4589 , piece_id=P3NL3JACSRBQ6BQZDMENVSFLHWEPVI47HRCFHUYI25QXLW2XBTRQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:22.141938960 (2h5m37.922872339s) [noise=true , address=77.90.31.13:4626 , piece_id=75CQFMO5CTU7RZDFQJ5U3FN3GGSTQAHYYF4EE4YDRBPBLGRJDVRA, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:22.740247145 (2h5m38.521180511s) [noise=true , address=77.90.31.19:4842 , piece_id=PSQZGLVQKUU3DMIOT3VQ62XZAFVLSGFQ66Z7PMBASX6KZPJ522IQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: <nil> Jul 26 23:57:50.538784063 (2h6m6.319717437s) [noise=true , address=77.90.31.17:4771 , piece_id=VVEBB6CJDW3CYLC6HM2MGCAGDXM7T5ZOHOMQXA6V3WBUI34Q3RQQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:44402->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538872737 (2h6m6.319806102s) [noise=true , address=104.36.82.6:4876 , piece_id=637YSFHNBIV4P5XWRCPH24LPJYY3M6MFFWC4Y5X6Z655ZZPGXJ6Q, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538916585 (2h6m6.319849954s) [noise=true , address=104.36.82.7:4912 , piece_id=GFNR5C5LGCJEOOWZ52RNSU2HHJPN5I3SRAAOIX3WSCYVGHIR5QQQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538961473 (2h6m6.319894837s) [noise=true , address=104.36.82.5:4837 , piece_id=UH67FWYQTAYDGTP65NA5BO2MSQIOSPGSJYYR6GYP55C7JR5EIGGQ, seg_pos={0 0}, upload=1927]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:36770->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:50.538990054 (2h6m6.319923431s) [seg_pos={0 0}, upload=1927]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:57:50.564064323 (2h6m6.344997692s) [upload=1927]: Done uploading segments... Jul 26 23:57:50.564195664 (2h6m6.345129035s) [upload=1927]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027222085s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X.X.X.......XXX...XX.....X.XXX...............X..X......X.] duration: 538.491978ms FAIL (2025-07-26 23:57:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.525264ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXX] duration: 30.030452728s FAIL (2025-07-26 23:57:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030499873s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX.X.X.X.......XXX...XX.....X.XXX...............X..X......X] duration: 30.00219189s FAIL (2025-07-26 23:57:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00222092s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 30.037657313s FAIL (2025-07-26 23:57:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== al tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902366849 (2h4m24.683300229s) [noise=true , address=104.36.82.5:4837 , piece_id=3QRZBC4262L2P3M5WRXYEE5QCB5QBTNWDPPYBMWGIZUBT3JNUWAA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902415935 (2h4m24.68334932s) [seg_pos={0 0}, upload=1901]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:56:08.929119953 (2h4m24.710053318s) [upload=1901]: Done uploading segments... Jul 26 23:56:08.929245899 (2h4m24.710179269s) [upload=1901]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:56:50.465037803 (2h5m6.245971187s) [upload=1919]: Starting upload Jul 26 23:56:50.465161980 (2h5m6.246095359s) [upload=1919]: Done starting upload Jul 26 23:56:50.465414494 (2h5m6.24634787s) [upload=1919]: Uploading segments... Jul 26 23:56:50.465592479 (2h5m6.246525864s) [upload=1919]: Got next segment. Inline: false Jul 26 23:56:50.465700612 (2h5m6.246633992s) [upload=1919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:56:50.465707959 (2h5m6.246641338s) [upload=1919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:56:50.498627491 (2h5m6.279560883s) [seg_pos={0 0}, upload=1919]: Begin upload segment... Jul 26 23:56:50.498707953 (2h5m6.27964135s) [seg_pos={0 0}, upload=1919]: Done begin upload segment. Jul 26 23:56:50.498742575 (2h5m6.279675962s) [noise=true , address=45.91.116.10:4082 , piece_id=NPWZ36AX4YZXZFWHIGDW63MY2MRF4UOAMZMPW5JWITMLMPA6LLFA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.499388568 (2h5m6.280321964s) [noise=true , address=77.90.31.21:4879 , piece_id=7APMPPEDNMZZLE3JFPCGDWRLTU7JZCKK7RI77HXHEZZ7JVABZZIQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.499992313 (2h5m6.280925701s) [noise=true , address=104.36.82.5:4839 , piece_id=TNVZP4Z4MCOP4MPKVXCUYDQFYJKYLYTJQ35HU4Y37FYFM6UQOWDQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.500649200 (2h5m6.281582589s) [noise=true , address=45.91.116.7:4009 , piece_id=7DGFXBBQ6NDDX2VXIDC3CZIC62QK6GSXODJ6SWQL7FTVCWSJD3CA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.501161510 (2h5m6.282094893s) [noise=true , address=77.90.31.19:4842 , piece_id=XERXEWTVI7AZ7CFETWVSSCDG7AAOK72DE7YNV6LR2EJNSANATZIA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.501647599 (2h5m6.282580985s) [noise=true , address=104.36.82.7:4911 , piece_id=CBS6NFPCFI6ENWBH6BEHIQ3UGCV5475GTLOSUZ4GYHQOP4XOS46Q, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.502106967 (2h5m6.283040356s) [noise=true , address=45.91.116.8:4047 , piece_id=YGJBQCFZMDVUWR7AXFOYVQWY7E3JKOTBBC3VFK6YIYL6I2QZMALQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.502504476 (2h5m6.283437864s) [noise=true , address=45.91.116.6:4975 , piece_id=LEB4XACHU3ZUCJ22DYAUBJ4UUQJUKXJTPBGSVE54A5KVVNXUSWLA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.502912525 (2h5m6.283845937s) [noise=true , address=104.36.82.4:4804 , piece_id=EFQ2RTSQRSU5ZRTRAFMJC3YHX4NG4EQHAHKDMME3JA4ZJI5PCJNA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.503298528 (2h5m6.284231926s) [noise=true , address=45.91.116.7:4010 , piece_id=5TPYUI2N5FK5MWIVJKDDIYU2YHG2LQIXM7LHGRKS4GA4QVSWBQIQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.503685240 (2h5m6.284618628s) [noise=true , address=45.91.116.4:4939 , piece_id=XJZOTMKIXTY5TTDHHJZHBYZHJVXDPMKWR7UOMLLCC7ABFTYMVPGQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.504060007 (2h5m6.284993397s) [noise=true , address=104.36.82.16:4294 , piece_id=GMBE2WSJIBVA2KNPVQGXIZY5ING5NMRBQLNAIJK3L5AEAHYK4DLA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.504460154 (2h5m6.285393542s) [noise=true , address=45.91.116.14:4189 , piece_id=ZRGTEUP4WOTY56VPDWI5TF4W2FT6KGLW67SID7KE2DZ57EBM4F2Q, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.504849562 (2h5m6.28578295s) [noise=true , address=77.90.31.12:4592 , piece_id=P2J5ODVVKQ4KM52YE4OSGF2THNSTTCY7TZTW6RGS3BO2Q6C4FLXA, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.505234196 (2h5m6.286167585s) [noise=true , address=77.90.31.14:4661 , piece_id=WHMJRFJQZMKNB7FHO3JPAVJ7IOHK4XVXX6JAZUQYJQVR6SXN6N6A, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.505589115 (2h5m6.286522504s) [noise=true , address=45.91.116.12:4225 , piece_id=U3W36JTBMJVYWOQRQHMAHGJ5IBY73N44B5W5YK7B3OC5AE65R5PQ, seg_pos={0 0}, upload=1919]: Uploading piece... Jul 26 23:56:50.611080234 (2h5m6.392013621s) [upload=1919]: Next returned nil segment Jul 26 23:56:50.611099550 (2h5m6.392032938s) [upload=1919]: Waiting for error group managing segments... Jul 26 23:56:50.614140486 (2h5m6.395073864s) [noise=true , address=45.91.116.4:4939 , piece_id=XJZOTMKIXTY5TTDHHJZHBYZHJVXDPMKWR7UOMLLCC7ABFTYMVPGQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.614630565 (2h5m6.395563942s) [noise=true , address=45.91.116.7:4010 , piece_id=5TPYUI2N5FK5MWIVJKDDIYU2YHG2LQIXM7LHGRKS4GA4QVSWBQIQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.615011656 (2h5m6.395945024s) [noise=true , address=45.91.116.10:4082 , piece_id=NPWZ36AX4YZXZFWHIGDW63MY2MRF4UOAMZMPW5JWITMLMPA6LLFA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.616327443 (2h5m6.397260815s) [noise=true , address=45.91.116.7:4009 , piece_id=7DGFXBBQ6NDDX2VXIDC3CZIC62QK6GSXODJ6SWQL7FTVCWSJD3CA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.616952065 (2h5m6.397885431s) [noise=true , address=77.90.31.14:4661 , piece_id=WHMJRFJQZMKNB7FHO3JPAVJ7IOHK4XVXX6JAZUQYJQVR6SXN6N6A, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.617571736 (2h5m6.398505113s) [noise=true , address=45.91.116.8:4047 , piece_id=YGJBQCFZMDVUWR7AXFOYVQWY7E3JKOTBBC3VFK6YIYL6I2QZMALQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.618080351 (2h5m6.399013728s) [noise=true , address=45.91.116.6:4975 , piece_id=LEB4XACHU3ZUCJ22DYAUBJ4UUQJUKXJTPBGSVE54A5KVVNXUSWLA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.618708942 (2h5m6.399642305s) [noise=true , address=45.91.116.12:4225 , piece_id=U3W36JTBMJVYWOQRQHMAHGJ5IBY73N44B5W5YK7B3OC5AE65R5PQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:50.782869602 (2h5m6.563802983s) [noise=true , address=45.91.116.14:4189 , piece_id=ZRGTEUP4WOTY56VPDWI5TF4W2FT6KGLW67SID7KE2DZ57EBM4F2Q, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:52.212986258 (2h5m7.993919628s) [noise=true , address=77.90.31.12:4592 , piece_id=P2J5ODVVKQ4KM52YE4OSGF2THNSTTCY7TZTW6RGS3BO2Q6C4FLXA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:56:52.748770297 (2h5m8.529703659s) [noise=true , address=77.90.31.19:4842 , piece_id=XERXEWTVI7AZ7CFETWVSSCDG7AAOK72DE7YNV6LR2EJNSANATZIA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: <nil> Jul 26 23:57:20.419852898 (2h5m36.200786278s) [noise=true , address=104.36.82.4:4804 , piece_id=EFQ2RTSQRSU5ZRTRAFMJC3YHX4NG4EQHAHKDMME3JA4ZJI5PCJNA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420263082 (2h5m36.20119646s) [noise=true , address=77.90.31.21:4879 , piece_id=7APMPPEDNMZZLE3JFPCGDWRLTU7JZCKK7RI77HXHEZZ7JVABZZIQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420279492 (2h5m36.201212856s) [noise=true , address=104.36.82.5:4839 , piece_id=TNVZP4Z4MCOP4MPKVXCUYDQFYJKYLYTJQ35HU4Y37FYFM6UQOWDQ, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420343148 (2h5m36.201276527s) [noise=true , address=104.36.82.16:4294 , piece_id=GMBE2WSJIBVA2KNPVQGXIZY5ING5NMRBQLNAIJK3L5AEAHYK4DLA, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420354433 (2h5m36.201287809s) [noise=true , address=104.36.82.7:4911 , piece_id=CBS6NFPCFI6ENWBH6BEHIQ3UGCV5475GTLOSUZ4GYHQOP4XOS46Q, seg_pos={0 0}, upload=1919]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:20.420374692 (2h5m36.201308072s) [seg_pos={0 0}, upload=1919]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:57:20.455540937 (2h5m36.236474325s) [upload=1919]: Done uploading segments... Jul 26 23:57:20.455569078 (2h5m36.236502457s) [upload=1919]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.037752922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.024663066s FAIL (2025-07-26 23:56:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== one begin upload segment. Jul 26 23:55:51.736607196 (2h4m7.51754058s) [noise=true , address=64.49.13.138:4482 , piece_id=QAMY2AP34W3YL4EWMFBI4PP5EB2QMCCMJ5RUBK6Y4YZKSKPDHH6Q, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737016986 (2h4m7.517950368s) [noise=true , address=104.36.82.47:4531 , piece_id=4V4W23MOJSCA5MVBLK3S47KTL4TCCISLQPSELV73THB7NQU25VHA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737265194 (2h4m7.51819857s) [noise=true , address=79.127.205.233:4640 , piece_id=PV6A4TGBW3XUVVM57YLHCD27KXZFNY2NMSJKSHMXVMLMI73FYPUQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737484924 (2h4m7.518418299s) [noise=true , address=104.36.82.54:4191 , piece_id=H4CQZ6G25HC62K73BF7PVYPNYLI34HVHOPZ4HXAP4TM6Q6GXQMCA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737709062 (2h4m7.51864244s) [noise=true , address=77.90.31.17:4784 , piece_id=QBKYDSDH4OOR2GGNBGNPNY5JQND5LRTJPKYID22KUW6CRSU6KWEQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.737932745 (2h4m7.518866126s) [noise=true , address=79.127.205.234:4614 , piece_id=QIXRNCPDPIV7K7SJS4XC3RPGTLYBM34EDL7OTX6TJVUCI6LSOI6Q, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738153974 (2h4m7.519087352s) [noise=true , address=79.127.219.38:4018 , piece_id=NSB4GJIUQ6XF7VPSP4RLR4YADNIXAVWIMFDVUKSSXVZFTH6G25BQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738372573 (2h4m7.519305954s) [noise=true , address=79.127.201.215:4359 , piece_id=4CAYQOW5UOSPDAQUGD7EXKNEJZLVSI5GDUWL5TW4AUGJ2WDRNNYQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738591768 (2h4m7.519525145s) [noise=true , address=45.91.116.9:4277 , piece_id=D6RHI7YTRLHQU7C3VYKQJSHPZLNPIHI7C7K6UHG6HRQVZFTK5ORQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.738826597 (2h4m7.519759978s) [noise=true , address=104.36.82.46:4484 , piece_id=OQTCIP4MUT4PAK2VT5AS4DMYDXDIK3HUKFL3CCEPS3NCCD44LM4Q, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739097434 (2h4m7.520030815s) [noise=true , address=77.90.31.10:4531 , piece_id=ZRDLEG5CWJCEXFZ346CQP6ZMSFBIVW4ZHZDTCR7TJZ6BOELZP5XQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739314856 (2h4m7.520248217s) [noise=true , address=79.127.220.98:4422 , piece_id=U4IQG7XK2WJ6YZRUKZYT5CRPO63QABH4WWVGUSJCMNCSCJHQANRA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739532946 (2h4m7.520466327s) [noise=true , address=104.36.82.52:4109 , piece_id=KZR6YMYHWEZSFLIA5SETSAPGYLRRTDB5IW6NLZRJN2NKO66F7WFA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739757421 (2h4m7.520690804s) [noise=true , address=104.36.82.53:4141 , piece_id=CVLR3VT4YOMO2E4FI42GQUAEPMZY5LT2OVOXCPMLIJM6U6PV7HIQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.739993694 (2h4m7.520927086s) [noise=true , address=77.90.31.13:4631 , piece_id=I7ZA4OA4UEGRPCZ5UKF2TPXC6YUADPZXR4YQPJBNDTDDAFYUPTUQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.740388551 (2h4m7.521321929s) [noise=true , address=64.49.13.138:4483 , piece_id=NM3MXV7YQ223YSH6SRD6UGMLFYSCPBOP7LC7PRTDRJYCHMPX4SWQ, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.740696228 (2h4m7.521629609s) [noise=true , address=77.90.31.21:4887 , piece_id=TVZAO6KLHQGPTNV3OGREIEPB3CIWXR4L6ZYTP443BSF47TM5Z5JA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.740983229 (2h4m7.521916608s) [noise=true , address=77.90.31.19:4871 , piece_id=4IFKVUY5NSUOQCGA54MFILJRBYTP7U47CMQJ56LMCLOYIOJ5PQFA, seg_pos={0 0}, upload=1906]: Uploading piece... Jul 26 23:55:51.809810613 (2h4m7.590743981s) [upload=1906]: Next returned nil segment Jul 26 23:55:51.809828216 (2h4m7.590761582s) [upload=1906]: Waiting for error group managing segments... Jul 26 23:55:51.813341620 (2h4m7.594274983s) [noise=true , address=79.127.219.38:4018 , piece_id=NSB4GJIUQ6XF7VPSP4RLR4YADNIXAVWIMFDVUKSSXVZFTH6G25BQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:52.334918607 (2h4m8.115851975s) [noise=true , address=79.127.205.234:4614 , piece_id=QIXRNCPDPIV7K7SJS4XC3RPGTLYBM34EDL7OTX6TJVUCI6LSOI6Q, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:52.701829381 (2h4m8.482762755s) [noise=true , address=79.127.205.233:4640 , piece_id=PV6A4TGBW3XUVVM57YLHCD27KXZFNY2NMSJKSHMXVMLMI73FYPUQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:53.063013429 (2h4m8.843946794s) [noise=true , address=77.90.31.13:4631 , piece_id=I7ZA4OA4UEGRPCZ5UKF2TPXC6YUADPZXR4YQPJBNDTDDAFYUPTUQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:54.328943719 (2h4m10.10987708s) [noise=true , address=79.127.201.215:4359 , piece_id=4CAYQOW5UOSPDAQUGD7EXKNEJZLVSI5GDUWL5TW4AUGJ2WDRNNYQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:54.663246223 (2h4m10.444179592s) [noise=true , address=77.90.31.19:4871 , piece_id=4IFKVUY5NSUOQCGA54MFILJRBYTP7U47CMQJ56LMCLOYIOJ5PQFA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:54.761753694 (2h4m10.54268706s) [noise=true , address=64.49.13.138:4483 , piece_id=NM3MXV7YQ223YSH6SRD6UGMLFYSCPBOP7LC7PRTDRJYCHMPX4SWQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:55:56.372494244 (2h4m12.15342761s) [noise=true , address=79.127.220.98:4422 , piece_id=U4IQG7XK2WJ6YZRUKZYT5CRPO63QABH4WWVGUSJCMNCSCJHQANRA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:55:59.102101660 (2h4m14.883035046s) [noise=true , address=64.49.13.138:4482 , piece_id=QAMY2AP34W3YL4EWMFBI4PP5EB2QMCCMJ5RUBK6Y4YZKSKPDHH6Q, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:56:00.743187561 (2h4m16.52412093s) [noise=true , address=45.91.116.9:4277 , piece_id=D6RHI7YTRLHQU7C3VYKQJSHPZLNPIHI7C7K6UHG6HRQVZFTK5ORQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: <nil> Jul 26 23:56:51.669324069 (2h5m7.450257455s) [noise=true , address=77.90.31.10:4531 , piece_id=ZRDLEG5CWJCEXFZ346CQP6ZMSFBIVW4ZHZDTCR7TJZ6BOELZP5XQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:56:51.669559077 (2h5m7.450492452s) [noise=true , address=104.36.82.53:4141 , piece_id=CVLR3VT4YOMO2E4FI42GQUAEPMZY5LT2OVOXCPMLIJM6U6PV7HIQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669616415 (2h5m7.450549779s) [noise=true , address=77.90.31.21:4887 , piece_id=TVZAO6KLHQGPTNV3OGREIEPB3CIWXR4L6ZYTP443BSF47TM5Z5JA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669665917 (2h5m7.450599293s) [noise=true , address=104.36.82.54:4191 , piece_id=H4CQZ6G25HC62K73BF7PVYPNYLI34HVHOPZ4HXAP4TM6Q6GXQMCA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: rpc: dial tcp 104.36.82.54:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669709557 (2h5m7.450642923s) [noise=true , address=104.36.82.46:4484 , piece_id=OQTCIP4MUT4PAK2VT5AS4DMYDXDIK3HUKFL3CCEPS3NCCD44LM4Q, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669717210 (2h5m7.450650577s) [noise=true , address=104.36.82.52:4109 , piece_id=KZR6YMYHWEZSFLIA5SETSAPGYLRRTDB5IW6NLZRJN2NKO66F7WFA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: write tcp 159.223.143.39:45574->104.36.82.52:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669774429 (2h5m7.450707805s) [noise=true , address=77.90.31.17:4784 , piece_id=QBKYDSDH4OOR2GGNBGNPNY5JQND5LRTJPKYID22KUW6CRSU6KWEQ, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: rpc: dial tcp 77.90.31.17:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:51.669898558 (2h5m7.450831937s) [noise=true , address=104.36.82.47:4531 , piece_id=4V4W23MOJSCA5MVBLK3S47KTL4TCCISLQPSELV73THB7NQU25VHA, seg_pos={0 0}, upload=1906]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: write tcp 159.223.143.39:50378->104.36.82.47:4531: use of closed network connection; context deadline exceeded Jul 26 23:56:51.669969401 (2h5m7.450902785s) [seg_pos={0 0}, upload=1906]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:56:51.692064813 (2h5m7.472998195s) [upload=1906]: Done uploading segments... Jul 26 23:56:51.692195106 (2h5m7.473128488s) [upload=1906]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.024683276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.X...X..X.......X...X....XX..............X.X........X...XX.] duration: 23.929279306s FAIL (2025-07-26 23:56:44 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.929299387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXX] duration: 30.036715565s FAIL (2025-07-26 23:56:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.036757719s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XX.X...X..X.......X...X....XX..............X.X........X...XX] duration: 1m0.001256143s FAIL (2025-07-26 23:56:20 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001308454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 30.031576958s FAIL (2025-07-26 23:56:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ose: conn closed: op Jul 26 23:55:38.792887535 (2h3m54.57382091s) [seg_pos={0 0}, upload=1891]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:55:38.815750378 (2h3m54.596683745s) [upload=1891]: Done uploading segments... Jul 26 23:55:38.815795701 (2h3m54.596729078s) [upload=1891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:55:38.943289758 (2h3m54.724223134s) [upload=1901]: Starting upload Jul 26 23:55:38.943363577 (2h3m54.724296944s) [upload=1901]: Done starting upload Jul 26 23:55:38.943438500 (2h3m54.724371867s) [upload=1901]: Uploading segments... Jul 26 23:55:38.943711796 (2h3m54.724645161s) [upload=1901]: Got next segment. Inline: false Jul 26 23:55:38.943745315 (2h3m54.724678679s) [upload=1901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:38.943750314 (2h3m54.724683689s) [upload=1901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:38.972810375 (2h3m54.753743783s) [seg_pos={0 0}, upload=1901]: Begin upload segment... Jul 26 23:55:38.972864526 (2h3m54.753797908s) [seg_pos={0 0}, upload=1901]: Done begin upload segment. Jul 26 23:55:38.972896174 (2h3m54.753829562s) [noise=true , address=45.91.116.13:4155 , piece_id=FAHGE7ZDRZGVP2QIXZR346CTB56RSL4V5RX3PUBPYMUQE6HFSWPQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.972924889 (2h3m54.753858283s) [noise=true , address=45.91.116.4:4940 , piece_id=2JM4LX2ZP35OI6TENM27UVJSKLTWAR5F3BN7BECXB4FJJNGCFQAQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973371247 (2h3m54.754304632s) [noise=true , address=45.91.116.6:4975 , piece_id=5ZHK2PU6U3PSDKSN63AFT7IF7QYMLGGULCEM5V3DSOU4E2VCAAHQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973440265 (2h3m54.754373647s) [noise=true , address=104.36.82.7:4909 , piece_id=OZ2OLA43OA6OVWHQFHDROJMCRMESPTQZN5ZBI4YEZ33KIIZYCQDQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973791860 (2h3m54.754725243s) [noise=true , address=45.91.116.9:4263 , piece_id=Z3LZWOD5AG7N3TEZVZFEUYI4DDCIKVBE24ZW3X37VQXYJLBQPTSA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.973838543 (2h3m54.754771927s) [noise=true , address=104.36.82.9:4005 , piece_id=M74RQT63CRR4G3FVH45K4OCFAFDV5CGCGYQHPRNLHLXSAUMQ4TMQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974147960 (2h3m54.755081344s) [noise=true , address=104.36.82.2:4731 , piece_id=YBX6D6PUX6ZB5QSQT6FKSTRHT3W62UEUPFUBCJSA5BTF4ERI5DCA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974245384 (2h3m54.755178766s) [noise=true , address=104.36.82.5:4837 , piece_id=3QRZBC4262L2P3M5WRXYEE5QCB5QBTNWDPPYBMWGIZUBT3JNUWAA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974462553 (2h3m54.755395938s) [noise=true , address=104.36.82.10:4079 , piece_id=M7KQW7PUVSTGLYDSFSK6R4ZT5M7GKBOBLKJOWX6JVUSQXK4DRWUA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974693632 (2h3m54.755627032s) [noise=true , address=45.91.116.7:4010 , piece_id=T4UPZISO5ZZUXH4NOHNLREZVON74NYFXT6P45BNKNOVGK52T23KQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.974763451 (2h3m54.755696827s) [noise=true , address=104.36.82.2:4730 , piece_id=BCS7UQ27PBQLBAJ6F6PTLRES5C3B4WHW73YIOERGLWUUAJIZRGTQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975102163 (2h3m54.756035544s) [noise=true , address=104.36.82.11:4114 , piece_id=LUKGOXOYPYHPBK63A2QMJUECNURLPE5THPH4DWMETCCDVTZKCL3A, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975244877 (2h3m54.756178259s) [noise=true , address=45.91.116.5:4298 , piece_id=JECAKQX4EUZ57QWUG5WUNH4DMXGATTS57CSSO5I2BREGTJCK375A, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975405189 (2h3m54.75633857s) [noise=true , address=77.90.31.14:4663 , piece_id=P6SPJFZIIQKBMLPBURURGINL7HMMSG2IJXQ7ETINRVYENPXACRQQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975634674 (2h3m54.75656806s) [noise=true , address=45.91.116.9:4262 , piece_id=DHP5JK2XWZDV7NP45CJYA57E7TFUDTXARNBASWTNGWJVK3OQW7DA, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:38.975703630 (2h3m54.756637006s) [noise=true , address=45.91.116.8:4045 , piece_id=V4RXNHVDSMTLIVCZMN4SDW7HLMWCCIOEUZZJ5ZX3J4RSYVUZ6OFQ, seg_pos={0 0}, upload=1901]: Uploading piece... Jul 26 23:55:39.067399969 (2h3m54.848333337s) [upload=1901]: Next returned nil segment Jul 26 23:55:39.067415508 (2h3m54.848348883s) [upload=1901]: Waiting for error group managing segments... Jul 26 23:55:39.071290081 (2h3m54.852223456s) [noise=true , address=45.91.116.13:4155 , piece_id=FAHGE7ZDRZGVP2QIXZR346CTB56RSL4V5RX3PUBPYMUQE6HFSWPQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.071840948 (2h3m54.852774317s) [noise=true , address=45.91.116.7:4010 , piece_id=T4UPZISO5ZZUXH4NOHNLREZVON74NYFXT6P45BNKNOVGK52T23KQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.072481061 (2h3m54.853414438s) [noise=true , address=45.91.116.4:4940 , piece_id=2JM4LX2ZP35OI6TENM27UVJSKLTWAR5F3BN7BECXB4FJJNGCFQAQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.072992218 (2h3m54.853925594s) [noise=true , address=45.91.116.5:4298 , piece_id=JECAKQX4EUZ57QWUG5WUNH4DMXGATTS57CSSO5I2BREGTJCK375A, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.073391224 (2h3m54.854324593s) [noise=true , address=45.91.116.6:4975 , piece_id=5ZHK2PU6U3PSDKSN63AFT7IF7QYMLGGULCEM5V3DSOU4E2VCAAHQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.073736742 (2h3m54.854670116s) [noise=true , address=77.90.31.14:4663 , piece_id=P6SPJFZIIQKBMLPBURURGINL7HMMSG2IJXQ7ETINRVYENPXACRQQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.073827343 (2h3m54.854760709s) [noise=true , address=45.91.116.8:4045 , piece_id=V4RXNHVDSMTLIVCZMN4SDW7HLMWCCIOEUZZJ5ZX3J4RSYVUZ6OFQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.504917220 (2h3m55.28585059s) [noise=true , address=45.91.116.9:4262 , piece_id=DHP5JK2XWZDV7NP45CJYA57E7TFUDTXARNBASWTNGWJVK3OQW7DA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:55:39.506696337 (2h3m55.287629708s) [noise=true , address=45.91.116.9:4263 , piece_id=Z3LZWOD5AG7N3TEZVZFEUYI4DDCIKVBE24ZW3X37VQXYJLBQPTSA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: <nil> Jul 26 23:56:08.902025840 (2h4m24.682959207s) [noise=true , address=104.36.82.11:4114 , piece_id=LUKGOXOYPYHPBK63A2QMJUECNURLPE5THPH4DWMETCCDVTZKCL3A, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902162029 (2h4m24.683095399s) [noise=true , address=104.36.82.2:4731 , piece_id=YBX6D6PUX6ZB5QSQT6FKSTRHT3W62UEUPFUBCJSA5BTF4ERI5DCA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 159.223.143.39:55760->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902193436 (2h4m24.683126824s) [noise=true , address=104.36.82.2:4730 , piece_id=BCS7UQ27PBQLBAJ6F6PTLRES5C3B4WHW73YIOERGLWUUAJIZRGTQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902268315 (2h4m24.683201687s) [noise=true , address=104.36.82.7:4909 , piece_id=OZ2OLA43OA6OVWHQFHDROJMCRMESPTQZN5ZBI4YEZ33KIIZYCQDQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902280088 (2h4m24.68321345s) [noise=true , address=104.36.82.9:4005 , piece_id=M74RQT63CRR4G3FVH45K4OCFAFDV5CGCGYQHPRNLHLXSAUMQ4TMQ, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902339682 (2h4m24.683273049s) [noise=true , address=104.36.82.10:4079 , piece_id=M7KQW7PUVSTGLYDSFSK6R4ZT5M7GKBOBLKJOWX6JVUSQXK4DRWUA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902366849 (2h4m24.683300229s) [noise=true , address=104.36.82.5:4837 , piece_id=3QRZBC4262L2P3M5WRXYEE5QCB5QBTNWDPPYBMWGIZUBT3JNUWAA, seg_pos={0 0}, upload=1901]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:56:08.902415935 (2h4m24.68334932s) [seg_pos={0 0}, upload=1901]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:56:08.929119953 (2h4m24.710053318s) [upload=1901]: Done uploading segments... Jul 26 23:56:08.929245899 (2h4m24.710179269s) [upload=1901]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.031598806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXX] duration: 30.025334716s FAIL (2025-07-26 23:56:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025352168s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX.X.X.X.......XXX...XX.....X.XXX...............X..X......] duration: 2.625398478s FAIL (2025-07-26 23:55:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.625448942s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.034451975s FAIL (2025-07-26 23:55:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 04 (2h3m7.392798688s) [seg_pos={0 0}, upload=1886]: Done begin upload segment. Jul 26 23:54:51.611900165 (2h3m7.392833551s) [noise=true , address=77.90.31.17:4780 , piece_id=MXK24WZJGMPX3THAEEJDCVCGBB2PX3TBFSM6MQ5H43VUV3RN243A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.611917856 (2h3m7.392851242s) [noise=true , address=104.36.82.53:4155 , piece_id=XN5QZ2S5UI32H4C6X7E6WOSCDS3WBNZ5DKDTKR2N2O4DHUKH4MPQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.612497958 (2h3m7.393431349s) [noise=true , address=79.127.201.214:4321 , piece_id=PEQ4RK3LDREIGED2ZZFUPRE57XH4KD3VHHKAW7C7FHYCZXEQW3TQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.612621971 (2h3m7.393555357s) [noise=true , address=79.127.205.245:4249 , piece_id=2GGDQGAWBTBUOQLUPLZU7FKD4WS7AKL6LNNOV5HCHE7RENRURQFA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.612948269 (2h3m7.393881652s) [noise=true , address=79.127.205.246:4278 , piece_id=LX2ENB6LMVMG2ZUKG32ZV2WZ5JKZPKHMCK6T62WBDRC73LFV6I6A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613213445 (2h3m7.394146832s) [noise=true , address=77.90.31.16:4765 , piece_id=6Z63YOOPGEH6JMXSVDR6KIAWZO7WXHOUTZXXWTZYEZVA5YDGNZSQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613341622 (2h3m7.394275006s) [noise=true , address=104.36.82.47:4542 , piece_id=7VATHDYL4G52ITO43MWNWPNL2X62AVSFVY765AX4B2AG5QCIHFLQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613671486 (2h3m7.394604868s) [noise=true , address=79.127.201.215:4343 , piece_id=LZSZ5QJGMGHRKHEQ24GVFLH2VZHAQZUTRV5VVCWO6SRV74JPZJSA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.613972571 (2h3m7.394905955s) [noise=true , address=104.36.82.54:4160 , piece_id=KLHY5NZ27H6R46SNVSOBHYH5G4CZ5GZFK2KTXZONX5PGPHZB4KTQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614275533 (2h3m7.395208921s) [noise=true , address=45.91.116.14:4206 , piece_id=U4GASVAEQZ7KP2LWZRANHZ4VEP5L3R2DE4GDURKB5766RYYHNXDA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614572378 (2h3m7.395505757s) [noise=true , address=79.127.220.101:4041 , piece_id=ANJNVQL6CGNZPDWTY3ZDZODH3EP35V7C7PK5LZZ65FJEJLJT322A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614869152 (2h3m7.395802533s) [noise=true , address=79.127.219.38:4015 , piece_id=RPQLDOSQMT7SNTRUQMPPEBMEDAKGN3P4KQCKPNEGS6HFUAL77WZA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.614936265 (2h3m7.395869656s) [noise=true , address=77.90.31.12:4620 , piece_id=XRP2ZHRMGUKHSMOQIGBXDJNOJF76EG3O3JENBOCERG34T2GBTSZA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.615461794 (2h3m7.396395185s) [noise=true , address=156.146.43.225:4434 , piece_id=CH2BF7FUDX35B56YEZY2HZBAG3ZLBXJZGIWSN7FCRNGWWWLIPLXA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.615945360 (2h3m7.396878746s) [noise=true , address=104.36.82.46:4510 , piece_id=EQOXPZZWQ33QSVFLX6IKL6RIHFLSVB2KBTE36DSVJNSNTZMJHFSA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.616461681 (2h3m7.397395068s) [noise=true , address=104.36.82.52:4106 , piece_id=4ZZSQQ7CQS3ZG7BLH7Y2WHGWI6GFPTIYBYSAIJXXNZ6TTRGZ74AQ, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.616899932 (2h3m7.397833318s) [noise=true , address=104.36.82.55:4225 , piece_id=A7FHVFHEE6IGOUUSZTOOLIIZQFWPMDANOZMBSZK2G5G4OUYDGIIA, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.617354692 (2h3m7.398288079s) [noise=true , address=77.90.31.13:4656 , piece_id=6JB5DZVWXCYOITFCTJ5C5QMUO4X5WDOTXYB2RHT2KSIFFL7L742A, seg_pos={0 0}, upload=1886]: Uploading piece... Jul 26 23:54:51.699422778 (2h3m7.480356141s) [upload=1886]: Next returned nil segment Jul 26 23:54:51.699445471 (2h3m7.480378858s) [upload=1886]: Waiting for error group managing segments... Jul 26 23:54:51.720281143 (2h3m7.501214521s) [noise=true , address=79.127.219.38:4015 , piece_id=RPQLDOSQMT7SNTRUQMPPEBMEDAKGN3P4KQCKPNEGS6HFUAL77WZA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:51.735111155 (2h3m7.516044525s) [noise=true , address=77.90.31.16:4765 , piece_id=6Z63YOOPGEH6JMXSVDR6KIAWZO7WXHOUTZXXWTZYEZVA5YDGNZSQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:52.117537914 (2h3m7.898471292s) [noise=true , address=156.146.43.225:4434 , piece_id=CH2BF7FUDX35B56YEZY2HZBAG3ZLBXJZGIWSN7FCRNGWWWLIPLXA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:52.208655786 (2h3m7.989589169s) [noise=true , address=79.127.205.245:4249 , piece_id=2GGDQGAWBTBUOQLUPLZU7FKD4WS7AKL6LNNOV5HCHE7RENRURQFA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:52.217561394 (2h3m7.998494757s) [noise=true , address=79.127.205.246:4278 , piece_id=LX2ENB6LMVMG2ZUKG32ZV2WZ5JKZPKHMCK6T62WBDRC73LFV6I6A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:53.480493157 (2h3m9.261426529s) [noise=true , address=77.90.31.12:4620 , piece_id=XRP2ZHRMGUKHSMOQIGBXDJNOJF76EG3O3JENBOCERG34T2GBTSZA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:54.314282140 (2h3m10.095215507s) [noise=true , address=79.127.201.214:4321 , piece_id=PEQ4RK3LDREIGED2ZZFUPRE57XH4KD3VHHKAW7C7FHYCZXEQW3TQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:54:54.958830009 (2h3m10.739763388s) [noise=true , address=45.91.116.14:4206 , piece_id=U4GASVAEQZ7KP2LWZRANHZ4VEP5L3R2DE4GDURKB5766RYYHNXDA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:59.695538751 (2h3m15.476472128s) [noise=true , address=79.127.220.101:4041 , piece_id=ANJNVQL6CGNZPDWTY3ZDZODH3EP35V7C7PK5LZZ65FJEJLJT322A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:55:18.516918083 (2h3m34.297851459s) [noise=true , address=79.127.201.215:4343 , piece_id=LZSZ5QJGMGHRKHEQ24GVFLH2VZHAQZUTRV5VVCWO6SRV74JPZJSA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:55:32.037831589 (2h3m47.818764959s) [noise=true , address=77.90.31.13:4656 , piece_id=6JB5DZVWXCYOITFCTJ5C5QMUO4X5WDOTXYB2RHT2KSIFFL7L742A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: <nil> Jul 26 23:55:51.546685461 (2h4m7.327618837s) [noise=true , address=104.36.82.47:4542 , piece_id=7VATHDYL4G52ITO43MWNWPNL2X62AVSFVY765AX4B2AG5QCIHFLQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547561359 (2h4m7.328494735s) [noise=true , address=104.36.82.53:4155 , piece_id=XN5QZ2S5UI32H4C6X7E6WOSCDS3WBNZ5DKDTKR2N2O4DHUKH4MPQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547651946 (2h4m7.328585323s) [noise=true , address=104.36.82.54:4160 , piece_id=KLHY5NZ27H6R46SNVSOBHYH5G4CZ5GZFK2KTXZONX5PGPHZB4KTQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: rpc: dial tcp 104.36.82.54:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547721911 (2h4m7.328655276s) [noise=true , address=77.90.31.17:4780 , piece_id=MXK24WZJGMPX3THAEEJDCVCGBB2PX3TBFSM6MQ5H43VUV3RN243A, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: write tcp 159.223.143.39:55998->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547789476 (2h4m7.328722851s) [noise=true , address=104.36.82.55:4225 , piece_id=A7FHVFHEE6IGOUUSZTOOLIIZQFWPMDANOZMBSZK2G5G4OUYDGIIA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547876105 (2h4m7.328809484s) [noise=true , address=104.36.82.46:4510 , piece_id=EQOXPZZWQ33QSVFLX6IKL6RIHFLSVB2KBTE36DSVJNSNTZMJHFSA, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: rpc: dial tcp 104.36.82.46:4510: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:51.547958673 (2h4m7.328892048s) [noise=true , address=104.36.82.52:4106 , piece_id=4ZZSQQ7CQS3ZG7BLH7Y2WHGWI6GFPTIYBYSAIJXXNZ6TTRGZ74AQ, seg_pos={0 0}, upload=1886]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:55:51.547975170 (2h4m7.32890854s) [seg_pos={0 0}, upload=1886]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:55:51.578843242 (2h4m7.359776623s) [upload=1886]: Done uploading segments... Jul 26 23:55:51.578899484 (2h4m7.359832864s) [upload=1886]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.034470426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 30.026066375s FAIL (2025-07-26 23:55:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 6 (2h3m24.604934395s) [upload=1891]: Starting upload Jul 26 23:55:08.824212724 (2h3m24.605146104s) [upload=1891]: Done starting upload Jul 26 23:55:08.824473693 (2h3m24.605407061s) [upload=1891]: Uploading segments... Jul 26 23:55:08.824676568 (2h3m24.605609945s) [upload=1891]: Got next segment. Inline: false Jul 26 23:55:08.824730302 (2h3m24.605663678s) [upload=1891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:08.824736974 (2h3m24.605670351s) [upload=1891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:08.859095743 (2h3m24.640029129s) [seg_pos={0 0}, upload=1891]: Begin upload segment... Jul 26 23:55:08.859160476 (2h3m24.640093846s) [seg_pos={0 0}, upload=1891]: Done begin upload segment. Jul 26 23:55:08.859191679 (2h3m24.640125061s) [noise=true , address=45.91.116.7:4011 , piece_id=776YHSW7RYC3ZVH4RPR62PRNE2UXNTUXMXLGTK5SVMDF5MJYZ5IQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.859215605 (2h3m24.640148988s) [noise=true , address=77.90.31.18:4806 , piece_id=VTS3CX7EZZGS726C5BROQOONCGYVTELIM7W6CWSNNV5CAFMJTTNQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.859693577 (2h3m24.640626967s) [noise=true , address=77.90.31.11:4553 , piece_id=ENYC7UKSC6FUGGEPG6JXOTD34LSVKD4KVCBTQTRRF2KWM5XR6VLQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.859727885 (2h3m24.640661274s) [noise=true , address=77.90.31.17:4770 , piece_id=SAZU4DIFGJ2U67ARSW3KQUSE43N6VY4AIHTM6FQIYSVV4225IVIA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860231888 (2h3m24.641165278s) [noise=true , address=104.36.82.11:4115 , piece_id=JNQPG3KE32EULC4SWTHMAKQ3BDRTTQNXBHP2CTYHBVTXF5T3V23Q, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860387364 (2h3m24.641320751s) [noise=true , address=77.90.31.12:4592 , piece_id=VCZHG7DNVFTC2TORJDOAWJFYRJL5CQY7765EOPIA57I2SDV2O56Q, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860638196 (2h3m24.641571582s) [noise=true , address=77.90.31.21:4878 , piece_id=7Y3ZGAVPQHAJBLBYXKMYQYMZ7VJHCKNR2HOMDMNOOXCUBNXJRCNA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.860884870 (2h3m24.641818256s) [noise=true , address=104.36.82.7:4910 , piece_id=4HAELEVHC4VBMHYS5ZUA222EZ5YZDEAVXJSKMBXO2ZVQSEHZYAEA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861014206 (2h3m24.641947585s) [noise=true , address=45.91.116.4:4939 , piece_id=6RE4PR4QNPUY5I75IQSV4A7AXBIFSYFD6VAVS3V5EXD5UOP7R5ZA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861309940 (2h3m24.642243322s) [noise=true , address=45.91.116.14:4189 , piece_id=SYK4K3YLSID5OEGIWEXYB2PYZMWPWHVS5IVM6I67CUZIQ5SDYYAQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861351351 (2h3m24.642284748s) [noise=true , address=45.91.116.7:4009 , piece_id=22FKJTPEWIPAW2S4NXW4OS3IJVTNO42SZRNYYUXTM33SJH2HCN6A, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861671962 (2h3m24.642605345s) [noise=true , address=104.36.82.8:4972 , piece_id=QX6VEOPKVL2YP4VG54LEQWAS4XXYXLSTTRUENYH2HYIZJ2D2VEQA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.861732627 (2h3m24.642666015s) [noise=true , address=77.90.31.18:4805 , piece_id=VCZO5OWKY2FBL6O5MERCGZYASNLEM7F4OUWNGB5T5TLHCSVE2LCA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.862026106 (2h3m24.642959494s) [noise=true , address=45.91.116.8:4048 , piece_id=XT44CG3QN7BSAMUGIT5DJJKQSSTSKGP2BVE653VILJC6EYDEGIFA, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.862169178 (2h3m24.643102556s) [noise=true , address=45.91.116.5:4298 , piece_id=VXXOTCYFMQ5YRFRTK3P7B42HZCMTRROXFHVEAN4LVQTWDLLLEVKQ, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.862622641 (2h3m24.643556028s) [noise=true , address=104.36.82.4:4803 , piece_id=HH3UBWRB22HFXVGZY2MZK4WT4TV5ZECSOBTYWDTCNDBMVS3U2K5Q, seg_pos={0 0}, upload=1891]: Uploading piece... Jul 26 23:55:08.951686160 (2h3m24.73261954s) [upload=1891]: Next returned nil segment Jul 26 23:55:08.951706185 (2h3m24.732639572s) [upload=1891]: Waiting for error group managing segments... Jul 26 23:55:08.955755607 (2h3m24.736688988s) [noise=true , address=45.91.116.4:4939 , piece_id=6RE4PR4QNPUY5I75IQSV4A7AXBIFSYFD6VAVS3V5EXD5UOP7R5ZA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.956456438 (2h3m24.737389801s) [noise=true , address=45.91.116.7:4011 , piece_id=776YHSW7RYC3ZVH4RPR62PRNE2UXNTUXMXLGTK5SVMDF5MJYZ5IQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.957551868 (2h3m24.738485241s) [noise=true , address=45.91.116.8:4048 , piece_id=XT44CG3QN7BSAMUGIT5DJJKQSSTSKGP2BVE653VILJC6EYDEGIFA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.958143228 (2h3m24.739076605s) [noise=true , address=45.91.116.5:4298 , piece_id=VXXOTCYFMQ5YRFRTK3P7B42HZCMTRROXFHVEAN4LVQTWDLLLEVKQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.958670840 (2h3m24.739604205s) [noise=true , address=45.91.116.7:4009 , piece_id=22FKJTPEWIPAW2S4NXW4OS3IJVTNO42SZRNYYUXTM33SJH2HCN6A, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.959966163 (2h3m24.740899531s) [noise=true , address=77.90.31.11:4553 , piece_id=ENYC7UKSC6FUGGEPG6JXOTD34LSVKD4KVCBTQTRRF2KWM5XR6VLQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:08.987506174 (2h3m24.768439551s) [noise=true , address=45.91.116.14:4189 , piece_id=SYK4K3YLSID5OEGIWEXYB2PYZMWPWHVS5IVM6I67CUZIQ5SDYYAQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:11.029206379 (2h3m26.810139755s) [noise=true , address=77.90.31.12:4592 , piece_id=VCZHG7DNVFTC2TORJDOAWJFYRJL5CQY7765EOPIA57I2SDV2O56Q, seg_pos={0 0}, upload=1891]: Done uploading piece. err: <nil> Jul 26 23:55:38.792571971 (2h3m54.573505332s) [noise=true , address=104.36.82.8:4972 , piece_id=QX6VEOPKVL2YP4VG54LEQWAS4XXYXLSTTRUENYH2HYIZJ2D2VEQA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792681069 (2h3m54.573614445s) [noise=true , address=104.36.82.4:4803 , piece_id=HH3UBWRB22HFXVGZY2MZK4WT4TV5ZECSOBTYWDTCNDBMVS3U2K5Q, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:53848->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792697299 (2h3m54.573630683s) [noise=true , address=104.36.82.11:4115 , piece_id=JNQPG3KE32EULC4SWTHMAKQ3BDRTTQNXBHP2CTYHBVTXF5T3V23Q, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:41412->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792746088 (2h3m54.573679457s) [noise=true , address=77.90.31.18:4806 , piece_id=VTS3CX7EZZGS726C5BROQOONCGYVTELIM7W6CWSNNV5CAFMJTTNQ, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792785701 (2h3m54.57371908s) [noise=true , address=77.90.31.18:4805 , piece_id=VCZO5OWKY2FBL6O5MERCGZYASNLEM7F4OUWNGB5T5TLHCSVE2LCA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:59332->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792800710 (2h3m54.573734077s) [noise=true , address=77.90.31.17:4770 , piece_id=SAZU4DIFGJ2U67ARSW3KQUSE43N6VY4AIHTM6FQIYSVV4225IVIA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 159.223.143.39:35684->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792835255 (2h3m54.573768623s) [noise=true , address=77.90.31.21:4878 , piece_id=7Y3ZGAVPQHAJBLBYXKMYQYMZ7VJHCKNR2HOMDMNOOXCUBNXJRCNA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:34806->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792850658 (2h3m54.573784028s) [noise=true , address=104.36.82.7:4910 , piece_id=4HAELEVHC4VBMHYS5ZUA222EZ5YZDEAVXJSKMBXO2ZVQSEHZYAEA, seg_pos={0 0}, upload=1891]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:41434->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:38.792887535 (2h3m54.57382091s) [seg_pos={0 0}, upload=1891]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:55:38.815750378 (2h3m54.596683745s) [upload=1891]: Done uploading segments... Jul 26 23:55:38.815795701 (2h3m54.596729078s) [upload=1891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026097447s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)