Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................X................] duration: 21.330167035s FAIL (2025-06-22 23:28:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.330195458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.......................................X...............] duration: 22.062969363s FAIL (2025-06-22 23:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.062991634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X...............X............................X.......] duration: 31.117611016s FAIL (2025-06-22 23:27:31 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 31.117633911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......................................X..............] duration: 15.796553016s FAIL (2025-06-22 23:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.796574233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X...............X............................X......] duration: 33.405660449s FAIL (2025-06-22 23:27:00 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 33.405685903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................................X.............] duration: 18.638183222s FAIL (2025-06-22 23:26:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.638209805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X...............X............................X.....] duration: 27.392766668s FAIL (2025-06-22 23:25:40 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 27.392787889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................................X............] duration: 13.219972569s FAIL (2025-06-22 23:25:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.220043516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X............................X....] duration: 13.426042469s FAIL (2025-06-22 23:24:40 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 13.426069491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.......................................X...........] duration: 20.924535333s FAIL (2025-06-22 23:24:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.924561173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X...............X............................X...] duration: 17.858595669s FAIL (2025-06-22 23:23:47 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 17.858642719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......................................X..........] duration: 29.684248382s FAIL (2025-06-22 23:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.684268159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X...............X............................X..] duration: 33.074176946s FAIL (2025-06-22 23:23:29 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 33.074220235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................................X.........] duration: 27.889769904s FAIL (2025-06-22 23:22:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.889795631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X...............X............................X.] duration: 17.583263558s FAIL (2025-06-22 23:22:19 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 17.583300998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X...............X............................X] duration: 1m0.008366579s FAIL (2025-06-22 23:22:01 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 1m0.008393311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................................X........] duration: 16.416932821s FAIL (2025-06-22 23:21:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.416989898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X...............X............................] duration: 27.527266978s FAIL (2025-06-22 23:20:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 27.527300065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................................X.......] duration: 35.561573303s FAIL (2025-06-22 23:20:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.561607572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X...............X...........................] duration: 24.672009407s FAIL (2025-06-22 23:19:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.672068115s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X.......................................X......] duration: 29.764852456s FAIL (2025-06-22 23:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.764880265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......................................X.....] duration: 10.523085787s FAIL (2025-06-22 23:18:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.523351333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............X..........................] duration: 21.90532804s FAIL (2025-06-22 23:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.905347386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X...............X.........................] duration: 42.209049734s FAIL (2025-06-22 23:18:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 42.20907726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......................................X....] duration: 30.265591093s FAIL (2025-06-22 23:18:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.265634135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............X........................] duration: 59.427196154s FAIL (2025-06-22 23:17:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 59.427222865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X.......................................X...] duration: 33.29416255s FAIL (2025-06-22 23:17:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.294186982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X................X.......................................X..] duration: 18.446971405s FAIL (2025-06-22 23:16:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.447012903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X................X.......................................X.] duration: 49.111708442s FAIL (2025-06-22 23:16:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.112465548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...............X.......................] duration: 15.499386109s FAIL (2025-06-22 23:16:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.49943558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...............X......................] duration: 37.921090352s FAIL (2025-06-22 23:15:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 37.92112327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X................X.......................................X] duration: 1m0.001413757s FAIL (2025-06-22 23:15:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001448681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...............X.....................] duration: 28.262181719s FAIL (2025-06-22 23:15:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.262232157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X................X.......................................] duration: 12.760808323s FAIL (2025-06-22 23:14:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.760828816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X................X......................................] duration: 28.328786962s FAIL (2025-06-22 23:14:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.328838371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............X....................] duration: 21.28511401s FAIL (2025-06-22 23:14:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.285144904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............X...................] duration: 16.165139991s FAIL (2025-06-22 23:13:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 16.165160824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............X..................] duration: 54.652012465s FAIL (2025-06-22 23:13:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 54.652048277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X................X.....................................] duration: 35.310124638s FAIL (2025-06-22 23:13:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.310141492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............X.................] duration: 20.970356658s FAIL (2025-06-22 23:11:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.97039349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X................X....................................] duration: 13.236488276s FAIL (2025-06-22 23:11:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.236515993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............X................] duration: 24.921131745s FAIL (2025-06-22 23:10:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.92117757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X................X...................................] duration: 15.423511038s FAIL (2025-06-22 23:10:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.423537417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X................X..................................] duration: 57.080198614s FAIL (2025-06-22 23:10:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.080217495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X...............X...............] duration: 30.395135264s FAIL (2025-06-22 23:09:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.395156063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...............X..............] duration: 19.078365342s FAIL (2025-06-22 23:09:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.078395402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X................X.................................] duration: 21.143319595s FAIL (2025-06-22 23:08:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.143345391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...............X.............] duration: 18.542578025s FAIL (2025-06-22 23:08:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.542612628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X................X................................] duration: 25.988200343s FAIL (2025-06-22 23:08:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.988229044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X................X...............................] duration: 20.684392411s FAIL (2025-06-22 23:07:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.684421523s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)