Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [......................X......XXXXXX.X...XX.X.X..............] duration: 238.312468ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:04:12 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 238.344346ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................XXXXXX.....XXXX.XXX..............] duration: 433.586792ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:04:04 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 433.61456ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX.............] duration: 660.032002ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:04:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 660.085038ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [......................X....X...XXX..XX.X..X..X.XX...........] duration: 338.263447ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:03:59 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 338.297157ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX............] duration: 854.119757ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:03:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 854.14663ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.......................X......XXXXXX.X...XX.X.X.............] duration: 358.500745ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:03:36 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 358.530025ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.......................X....X...XXX..XX.X..X..X.XX..........] duration: 254.342221ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:03:36 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 254.373865ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................XXXXXX.....XXXX.XXX.............] duration: 513.982759ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:03:35 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 514.02139ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 316.541445ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:03:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 316.569944ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......................XX.....X.XXXXXX.X..XXX...............] duration: 1.700829265s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:03:31 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.700859218s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................XX.....X.XXXXXX.X..XXX..............] duration: 1.768986975s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:02:56 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.769177622s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX...........] duration: 656.758856ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:02:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 656.801776ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XXXXXX.....XXXX.XXX............] duration: 514.120408ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:02:44 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 514.164938ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [........................X......XXXXXX.X...XX.X.X............] duration: 310.420409ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:02:35 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 310.44427ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 200.229438ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:02:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 200.273799ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [........................X....X...XXX..XX.X..X..X.XX.........] duration: 248.892377ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:02:31 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 248.926627ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................XX.....X.XXXXXX.X..XXX.............] duration: 1.822478004s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:02:21 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.822514432s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.........................X......XXXXXX.X...XX.X.X...........] duration: 264.661025ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:02:16 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 264.689986ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............................XXXXXX.....XXXX.XXX...........] duration: 465.925825ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:02:09 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 465.950835ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 200.363117ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:01:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 200.410442ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.........................X....X...XXX..XX.X..X..X.XX........] duration: 257.178723ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:01:45 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 257.218797ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX..........] duration: 750.839322ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:01:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 750.865952ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 272.760139ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:01:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 272.792813ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX.........] duration: 632.693418ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:01:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 632.720367ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................XXXXXX.....XXXX.XXX..........] duration: 505.348708ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:01:24 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 505.384577ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................XX.....X.XXXXXX.X..XXX............] duration: 1.762872806s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:01:18 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.762903449s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..........................X......XXXXXX.X...XX.X.X..........] duration: 275.8369ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:01:06 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 275.864229ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..........................X....X...XXX..XX.X..X..X.XX.......] duration: 269.095458ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:01:03 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 269.123507ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 213.879332ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:00:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 213.917255ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...........................X......XXXXXX.X...XX.X.X.........] duration: 242.118401ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:00:35 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 242.160266ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX........] duration: 511.655225ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:00:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.695807ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [................................XXXXXX.....XXXX.XXX.........] duration: 622.590157ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:00:15 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 622.781817ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................XX.....X.XXXXXX.X..XXX...........] duration: 1.689332657s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:00:15 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.68937144s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...........................X....X...XXX..XX.X..X..X.XX......] duration: 272.90128ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:00:15 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 272.936873ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 217.165985ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:00:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 217.235583ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX.......] duration: 727.892227ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 21:59:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 727.922151ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.................................XXXXXX.....XXXX.XXX........] duration: 443.006428ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 21:59:54 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 443.037231ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 209.548669ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:59:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 209.578521ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................XX.....X.XXXXXX.X..XXX..........] duration: 1.633358892s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 21:59:31 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.633449204s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..................................XXXXXX.....XXXX.XXX.......] duration: 450.277203ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 21:59:27 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 450.324159ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [............................X......XXXXXX.X...XX.X.X........] duration: 401.205815ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 21:59:23 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 401.236018ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [............................X....X...XXX..XX.X..X..X.XX.....] duration: 260.329146ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 21:59:23 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 260.416054ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XX.....X.XXXXXX.X..XXX.........] duration: 1.701774s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 21:59:11 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.701807874s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...................................XXXXXX.....XXXX.XXX......] duration: 463.836066ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 21:59:09 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 463.881096ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 257.633473ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:58:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 257.663625ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....................................XXXXXX.....XXXX.XXX.....] duration: 418.135592ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 21:58:48 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 418.165808ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX......] duration: 1.341487162s trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 21:58:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.341518833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX.....] duration: 454.732575ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 21:58:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 454.755006ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............................XX.....X.XXXXXX.X..XXX........] duration: 1.720290569s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 21:58:18 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.720316442s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.............................X......XXXXXX.X...XX.X.X.......] duration: 242.890121ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 21:58:10 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 242.914745ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)