Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X..........................X........] duration: 16.323950102s FAIL (2025-06-23 01:03:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.323977571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..........................X.......] duration: 11.955102602s FAIL (2025-06-23 01:03:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.955140716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.................................................X.....] duration: 10.003584783s FAIL (2025-06-23 01:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.003633198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..........................X......] duration: 9.400329854s FAIL (2025-06-23 01:02:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.400358957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................................................X....] duration: 22.925469954s FAIL (2025-06-23 01:02:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.925490798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..........................X.....] duration: 14.511146423s FAIL (2025-06-23 01:02:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.511171413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.................................................X...] duration: 23.459457107s FAIL (2025-06-23 01:02:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.459497063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................................................X..] duration: 36.133138576s FAIL (2025-06-23 01:01:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.133163905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X..........................X....] duration: 10.841676934s FAIL (2025-06-23 01:00:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.841726038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.................................................X.] duration: 18.287300325s FAIL (2025-06-23 01:00:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.287354121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..........................X...] duration: 29.289293138s FAIL (2025-06-23 01:00:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.289343252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................................................X] duration: 1m0.001049255s FAIL (2025-06-23 01:00:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001074491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..........................X..] duration: 21.079605158s FAIL (2025-06-23 00:59:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.079632656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..........................X.] duration: 36.880751364s FAIL (2025-06-23 00:58:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.880772374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.879133885s FAIL (2025-06-23 00:58:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.879156383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 34.019988291s FAIL (2025-06-23 00:58:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.020005019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X..........................X] duration: 1m0.029213334s FAIL (2025-06-23 00:58:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.029245431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 38.664223327s FAIL (2025-06-23 00:57:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.664251001s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 9.095809993s FAIL (2025-06-23 00:56:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.095824694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 32.173305175s FAIL (2025-06-23 00:56:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.17333073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 14.052252659s FAIL (2025-06-23 00:56:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.052280342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 29.998056138s FAIL (2025-06-23 00:55:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.99807015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.559523633s FAIL (2025-06-23 00:55:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.559554372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 20.890629777s FAIL (2025-06-23 00:55:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.890649299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 44.145407984s FAIL (2025-06-23 00:54:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.145437815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 30.449908706s FAIL (2025-06-23 00:54:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.449931808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 19.038243887s FAIL (2025-06-23 00:54:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.038271526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 13.826564951s FAIL (2025-06-23 00:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.826621955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 13.244701024s FAIL (2025-06-23 00:53:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.24472873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 27.154973791s FAIL (2025-06-23 00:53:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.155002411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 19.791130763s FAIL (2025-06-23 00:52:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.791155841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 39.091096975s FAIL (2025-06-23 00:52:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 39.091128649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 13.016020769s FAIL (2025-06-23 00:52:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.016035043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 19.181974978s FAIL (2025-06-23 00:51:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.183223418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 19.48852311s FAIL (2025-06-23 00:51:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.488553224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 9.703005488s FAIL (2025-06-23 00:50:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.703040101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.841607308s FAIL (2025-06-23 00:50:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.841633972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 10.62805439s FAIL (2025-06-23 00:50:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.628086441s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 22.11064595s FAIL (2025-06-23 00:50:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.110675512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 23.474358745s FAIL (2025-06-23 00:49:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.474378405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 32.779106823s FAIL (2025-06-23 00:49:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.779133535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X........................X..................................] duration: 21.280340749s FAIL (2025-06-23 00:49:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.280405564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 25.767702086s FAIL (2025-06-23 00:48:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.767735564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X........................X.................................] duration: 17.614824509s FAIL (2025-06-23 00:48:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.614843437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.X........................X................................] duration: 41.751697863s FAIL (2025-06-23 00:47:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.751721216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 24.85742445s FAIL (2025-06-23 00:47:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.857445231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 34.857950544s FAIL (2025-06-23 00:47:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.85798115s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.X........................X...............................] duration: 35.359224539s FAIL (2025-06-23 00:47:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.359354926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.X........................X..............................] duration: 10.377227168s FAIL (2025-06-23 00:45:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.377249452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 9.528640218s FAIL (2025-06-23 00:45:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 056603249s) [upload=83370]: Done uploading segments... Jun 23 00:36:29.493306521 (104h13m5.056629518s) [upload=83370]: Upload finished. err: <nil> Jun 23 00:36:43.017010067 (104h13m18.580333072s) [upload=83378]: Starting upload Jun 23 00:36:43.017104046 (104h13m18.580427053s) [upload=83378]: Done starting upload Jun 23 00:36:43.017465013 (104h13m18.580788006s) [upload=83378]: Uploading segments... Jun 23 00:36:43.017522996 (104h13m18.580846007s) [upload=83378]: Got next segment. Inline: false Jun 23 00:36:43.017562228 (104h13m18.580885231s) [upload=83378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:36:43.017567520 (104h13m18.580890525s) [upload=83378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:36:43.055958752 (104h13m18.619281767s) [seg_pos={0 0}, upload=83378]: Begin upload segment... Jun 23 00:36:43.056025403 (104h13m18.619348395s) [seg_pos={0 0}, upload=83378]: Done begin upload segment. Jun 23 00:36:43.056061259 (104h13m18.619384273s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056163298 (104h13m18.619486302s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056639271 (104h13m18.619962295s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.056721014 (104h13m18.62004402s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057223522 (104h13m18.620546544s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057364183 (104h13m18.620687191s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057750813 (104h13m18.621073827s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.057871481 (104h13m18.621194471s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058146325 (104h13m18.621469335s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058387586 (104h13m18.621710595s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058549083 (104h13m18.621872163s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058847603 (104h13m18.62217061s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.058948198 (104h13m18.622271211s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059287966 (104h13m18.622610967s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059440829 (104h13m18.622763839s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059677790 (104h13m18.623000796s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.059823761 (104h13m18.623146769s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.060339567 (104h13m18.62366259s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Uploading piece... Jun 23 00:36:43.144991762 (104h13m18.708314768s) [upload=83378]: Next returned nil segment Jun 23 00:36:43.145016988 (104h13m18.708340001s) [upload=83378]: Waiting for error group managing segments... Jun 23 00:36:43.159614490 (104h13m18.722937501s) [noise=true , address=79.127.220.98:4429 , piece_id=QU7E7RMNZFYTIHMW2FIYUGOYT2FJSLHDZROEZ6K4VFH6LQF2DUOA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:36:43.218074113 (104h13m18.781397114s) [noise=true , address=79.127.201.214:4304 , piece_id=SQY2EYDIOUIMQ7MQNLVCTKR3NVG3ZMA2SC5JDUO6WJIUBIY7VMUQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12oByme3wstnmNDGAYFygy3KCUhK7gAoPa9k7cHKiCoSFmDiYtH, address:79.127.201.214:4304): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:43.256685465 (104h13m18.820008471s) [noise=true , address=77.90.31.14:4674 , piece_id=BPACLD5VPX66FQDD35I2Y7IMIGI5RVQOTTDVMIU3LZECANSG54TA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:43.271190816 (104h13m18.834513821s) [noise=true , address=77.90.31.18:4825 , piece_id=V4YL2AJ42OY7YQKPLN3KO6CQB3Q2D7KURLE5HPJV566SR6NKDXPQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:44.385018041 (104h13m19.948341041s) [noise=true , address=104.36.82.53:4124 , piece_id=LTYADN4URM6LGDTX7PLRX6QRGJ42KPRQQ7NTLSQPJ7HMFJ4QRTIA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:44.983620490 (104h13m20.546943484s) [noise=true , address=104.36.82.40:6976 , piece_id=LRSZJ25OYBIGPKHSMLJSHSGTWHA7R5LMGEA2G77XEFUWJKV32P2A, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.008021921 (104h13m20.571344907s) [noise=true , address=104.36.82.5:5843 , piece_id=HCS6FLLKDP57MXYVJNMBFVLLFLQSU3YV2LPKVDNLSDA2H3IMVQHA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:45.048117745 (104h13m20.611440757s) [noise=true , address=77.90.31.11:4578 , piece_id=F4N4CIMACIEED6FB3D2ZDFBVBCXWLKZCSQS6MJN47Q5B5WUDEAIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 23 00:36:46.999765197 (104h13m22.563088199s) [noise=true , address=45.91.116.10:5084 , piece_id=2MDWF5K525CQTVTUR7GGGGQO4JCPTCCU5PA36NETWC2DVIIPNIMQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:48.635720064 (104h13m24.199043056s) [noise=true , address=104.36.82.41:6781 , piece_id=J34DPQAPREFCBLKUJKK5V7L4Y4YEBKBI2E6GVG776IVUL2UKEYFA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:50.782337090 (104h13m26.345660094s) [noise=true , address=104.36.82.20:6453 , piece_id=ILGGUNSX6QEA2ZGSJBRRA2IBSURRIV5HXGWPQ2IF2OHUYHRUCCWA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:51.303798054 (104h13m26.867121062s) [noise=true , address=79.127.219.38:4025 , piece_id=V46MDHHDCZQ7SNRYCWLEFS2CTTP324QXPGMMADXBEMHXXVDNQZYQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:52.112477337 (104h13m27.67580034s) [noise=true , address=77.90.31.16:4746 , piece_id=JIBDFSOBTJPDQDZOSNS3DCEC5XSQBTACBZET3GKG2T4VMGOWZNJQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:36:55.156608702 (104h13m30.719931696s) [noise=true , address=104.36.82.39:6737 , piece_id=4OXVR4UO7ZUWISPT4ZGNDFQFBRZR74CYP3RTR7S5N23OTIQOKGCQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:02.691823632 (104h13m38.255146622s) [noise=true , address=104.36.82.13:6208 , piece_id=A3BRSE7B4H2S5DBAGA75BNCAVK45IG37PEDQTNKH76IEHJFSFHUA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:03.139182441 (104h13m38.702505428s) [noise=true , address=104.36.82.56:4242 , piece_id=DKYBOQQJQADD3VYEWRMGI4DDTEZBMUXOTY3OIRG5C37LU6Z2YRIQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:10.822000644 (104h13m46.385323646s) [noise=true , address=104.36.82.52:4117 , piece_id=OG34KMCCU26SFLGUII2WG7ALE74ZB2EHVQ52YTMJWUIXW6QSZBQQ, seg_pos={0 0}, upload=83378]: Done uploading piece. err: <nil> Jun 23 00:37:42.963083703 (104h14m18.526406706s) [noise=true , address=104.36.82.11:6141 , piece_id=3HFK7SQCSEKER3W4VJOLBGJ3GLOVFFGE37LECFPEZT4YLGNHSCTA, seg_pos={0 0}, upload=83378]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:6141): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:37:42.963157967 (104h14m18.526480971s) [seg_pos={0 0}, upload=83378]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:37:42.986482481 (104h14m18.549805486s) [upload=83378]: Done uploading segments... Jun 23 00:37:42.986517390 (104h14m18.549840396s) [upload=83378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.52867861s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)