Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 26.988881025s FAIL (2025-06-21 23:31:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.98891323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 17.921332963s FAIL (2025-06-21 23:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.921369864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 20.095477359s FAIL (2025-06-21 23:30:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.095511896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 21.593312669s FAIL (2025-06-21 23:30:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.593344034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 11.683998448s FAIL (2025-06-21 23:29:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.684032894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 23.252519647s FAIL (2025-06-21 23:29:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.252548558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 12.327189305s FAIL (2025-06-21 23:28:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.3272192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 37.538319581s FAIL (2025-06-21 23:28:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.538358512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 27.721085477s FAIL (2025-06-21 23:28:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.7211121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 36.039970486s FAIL (2025-06-21 23:27:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.040003206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 17.957641253s FAIL (2025-06-21 23:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.957667993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 20.851017148s FAIL (2025-06-21 23:27:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.851041932s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 21.517092185s FAIL (2025-06-21 23:26:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.517118897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 13.415120151s FAIL (2025-06-21 23:26:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.415140196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 22.944683705s FAIL (2025-06-21 23:26:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.944734871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 30.3278083s FAIL (2025-06-21 23:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.327836848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 20.199914384s FAIL (2025-06-21 23:25:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.199937276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 19.989195261s FAIL (2025-06-21 23:24:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.989213065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 15.625490489s FAIL (2025-06-21 23:24:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.625513664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 41.684243914s FAIL (2025-06-21 23:23:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 41.684277874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 21.652171163s FAIL (2025-06-21 23:22:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.652290989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 19.193794406s FAIL (2025-06-21 23:22:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.193930732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 16.591023922s FAIL (2025-06-21 23:22:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.591047872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 17.823036061s FAIL (2025-06-21 23:22:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.823061765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 19.349440468s FAIL (2025-06-21 23:21:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.349465746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 22.694634432s FAIL (2025-06-21 23:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.694653957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 32.752791674s FAIL (2025-06-21 23:21:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.752824565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 15.302464009s FAIL (2025-06-21 23:20:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.302487567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 38.619531998s FAIL (2025-06-21 23:20:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.619569043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 23.615204838s FAIL (2025-06-21 23:19:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.615230768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 28.154200672s FAIL (2025-06-21 23:19:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.154230981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 17.019624714s FAIL (2025-06-21 23:19:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.019644565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 18.061670085s FAIL (2025-06-21 23:18:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.061699958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 15.40219229s FAIL (2025-06-21 23:17:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.402301559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 9.609190021s FAIL (2025-06-21 23:17:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.609220733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 17.863932129s FAIL (2025-06-21 23:17:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.863961959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 19.896219628s FAIL (2025-06-21 23:17:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.896245327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 19.886066002s FAIL (2025-06-21 23:16:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.886155736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 26.784269064s FAIL (2025-06-21 23:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.78429127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 16.117015853s FAIL (2025-06-21 23:15:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.117046069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001555515s FAIL (2025-06-21 23:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001834569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 21.193962436s FAIL (2025-06-21 23:14:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.193988278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 15.660644898s FAIL (2025-06-21 23:13:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.660672774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 29.436523255s FAIL (2025-06-21 23:12:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.436554707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 39.74237039s FAIL (2025-06-21 23:12:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 39.742397676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 21.275606955s FAIL (2025-06-21 23:10:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.275648374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 24.43334002s FAIL (2025-06-21 23:10:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.4333708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 14.143453224s FAIL (2025-06-21 23:09:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.143482877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 20.09873975s FAIL (2025-06-21 23:08:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.098769986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 16.60659723s FAIL (2025-06-21 23:07:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 2994]: Done starting upload Jun 21 23:03:12.157295492 (78h39m47.720618482s) [upload=62994]: Uploading segments... Jun 21 23:03:12.157569421 (78h39m47.720892411s) [upload=62994]: Got next segment. Inline: false Jun 21 23:03:12.157636334 (78h39m47.720959333s) [upload=62994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 23:03:12.157641283 (78h39m47.720964282s) [upload=62994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 23:03:12.194202426 (78h39m47.757525429s) [seg_pos={0 0}, upload=62994]: Begin upload segment... Jun 21 23:03:12.194254676 (78h39m47.757577663s) [seg_pos={0 0}, upload=62994]: Done begin upload segment. Jun 21 23:03:12.194295158 (78h39m47.757618173s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194316742 (78h39m47.757639749s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.194749003 (78h39m47.758072012s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195040779 (78h39m47.758363779s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195268477 (78h39m47.758591484s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195482306 (78h39m47.758805305s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195569014 (78h39m47.758892014s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195908034 (78h39m47.759231036s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.195914241 (78h39m47.759237228s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196243917 (78h39m47.759566925s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196322865 (78h39m47.759645875s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196564228 (78h39m47.759887233s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196694834 (78h39m47.76001784s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.196852214 (78h39m47.760175203s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197091627 (78h39m47.760414634s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197123848 (78h39m47.760446846s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197430553 (78h39m47.760753554s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.197732318 (78h39m47.76105532s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Uploading piece... Jun 21 23:03:12.282624807 (78h39m47.845947808s) [upload=62994]: Next returned nil segment Jun 21 23:03:12.282644177 (78h39m47.845967177s) [upload=62994]: Waiting for error group managing segments... Jun 21 23:03:12.301500025 (78h39m47.864823031s) [noise=true , address=104.36.82.56:4239 , piece_id=7RJYHYTB5G4PMHJOHXLGPVRT7KWGEGW37VJZCPGEYC7DVJL22IEQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.338560176 (78h39m47.901883186s) [noise=true , address=104.36.82.54:4180 , piece_id=37IFBK5U34INHSY4WI2QEEQ47KEYA2LAEESIJ5RB2JQVT55DCEAA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:12.353567724 (78h39m47.916890712s) [noise=true , address=77.90.31.14:4666 , piece_id=WWSU4L7AKMX7SCHWUA5SGXRLJB57NDDLMTS32XIOWXW5MVRCVMGA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.425010002 (78h39m47.988333014s) [noise=true , address=45.91.116.12:5249 , piece_id=EOH6NQ5X2BF2XGSXEP32SYO5SPPRCL5GCYNVLBSKMLF3DFMCJAOQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.559667339 (78h39m48.122990329s) [noise=true , address=156.146.43.225:4467 , piece_id=25JXSBXRYBJBB7MB3WX2IHBWG7WRQ3HFKQMM57FAT5XCNKJLLSWQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:12.824470485 (78h39m48.387793479s) [noise=true , address=156.146.43.225:4454 , piece_id=6MJ66QUOQVCGK3FHZWZJ2OKI46BFI5RW6VVXMLCHE6LUAPCZJCRA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.036351387 (78h39m48.599674377s) [noise=true , address=104.36.82.9:6033 , piece_id=4FGPCQI5X22YK5RASONBYUQKFD4SGZDSNZOCLBZUPR5CNSUZZ5QA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.146623036 (78h39m48.709946021s) [noise=true , address=79.127.201.214:4320 , piece_id=NIHXDJYDWM7RF2YY75ND5CKYH2O734LUMACQGZRDM3RMR4PA6UHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.560445551 (78h39m49.123768537s) [noise=true , address=77.90.31.11:4577 , piece_id=ZTMKZBEHFTJVTFND4T367IP7KPP4TKHUYU3TAKN3DUHEWK2LSLWA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: storage node overloaded, request limit: 2000 Jun 21 23:03:13.579450750 (78h39m49.142773744s) [noise=true , address=104.36.82.5:5854 , piece_id=RAUPN2FWEC47DYZWY77JVIBS5DHGRQTRGJKEK7LCSYTLZZWPZFPQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:13.724129379 (78h39m49.287452369s) [noise=true , address=104.36.82.10:6100 , piece_id=WGS55TAVUAIIE5KTFYKG6RB2E3DMUX75463F3MPLVAHNIOQGEIHA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:14.780062664 (78h39m50.343385651s) [noise=true , address=104.36.82.16:6296 , piece_id=74WDC2WMBY62PSDBYUQ5WROBP2NNXBNEU75MLQXF5XYDROWGWQTA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:18.822633893 (78h39m54.385956894s) [noise=true , address=77.90.31.19:4845 , piece_id=GRF7FZK6GAEQKNIARYQWYYVQQGIGVNDCTI2ZG6YDRPET74V56QZQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:20.585479531 (78h39m56.148802534s) [noise=true , address=104.36.82.20:6451 , piece_id=GJAF5I74K6RXJT3ETJCJW5ZIDMDJR5VTCSBJ54V7ESM2B2H2YAVQ, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:25.391821780 (78h40m0.955144788s) [noise=true , address=104.36.82.52:4096 , piece_id=56ZOGTT7K6OA574H4KNCEKNDCRQLLOXRH2QV5C3W63SWXQ5S4C4A, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:03:39.508660492 (78h40m15.071983482s) [noise=true , address=104.36.82.53:4147 , piece_id=GCRYVKMOWUJZZIBR4XTNGZHWV4OFI3RGFKLBAOQ6YTI6W37CNFSA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: <nil> Jun 21 23:04:12.128013650 (78h40m47.691336649s) [noise=true , address=104.36.82.55:4212 , piece_id=P7ANT6LIUO45NSQYFNPIDRISCOPRUWFXS3KDJEZDQ6BUANV2QCQA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: read tcp 159.223.143.39:47784->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 21 23:04:12.128053405 (78h40m47.691376416s) [noise=true , address=104.36.82.12:6166 , piece_id=IPE4ZGDPLAVKLCAFBNKOCWM6DGML3WZ242BX7YG2BQYXQ5F4LDJA, seg_pos={0 0}, upload=62994]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:6166): protocol: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56704->104.36.82.12:6166: use of closed network connection Jun 21 23:04:12.128108814 (78h40m47.691431821s) [seg_pos={0 0}, upload=62994]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 23:04:12.152878890 (78h40m47.716201883s) [upload=62994]: Done uploading segments... Jun 21 23:04:12.152914585 (78h40m47.716237576s) [upload=62994]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.60661625s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)