Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX.....................] duration: 711.006498ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:10:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 711.036554ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...................XXXXXX.....XXXX.XXX......................] duration: 628.642413ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:10:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 628.671129ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..............X....X...XXX..XX.X..X..X.XX...................] duration: 662.058857ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:10:20 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 662.085992ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.............X......XXXXXX.X...XX.X.X.......................] duration: 319.810483ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:10:16 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 319.841595ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...............X....X...XXX..XX.X..X..X.XX..................] duration: 345.326931ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:10:01 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 345.449743ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 272.18149ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:09:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 272.212218ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............XX.....X.XXXXXX.X..XXX........................] duration: 1.783467952s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:09:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.783504029s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....................XXXXXX.....XXXX.XXX.....................] duration: 529.825268ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:09:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 529.863136ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX....................] duration: 564.569851ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:09:26 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 564.602787ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 245.462058ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:09:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 245.491773ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [................X....X...XXX..XX.X..X..X.XX.................] duration: 316.123216ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:09:18 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 316.199743ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..............X......XXXXXX.X...XX.X.X......................] duration: 311.114013ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:09:14 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 311.136873ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX...................] duration: 752.547177ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:09:10 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 752.581037ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....................XXXXXX.....XXXX.XXX....................] duration: 489.989815ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:08:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 490.012514ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...............X......XXXXXX.X...XX.X.X.....................] duration: 259.765058ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:08:48 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 259.863294ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............XX.....X.XXXXXX.X..XXX.......................] duration: 1.729978729s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:08:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.730007996s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......................XXXXXX.....XXXX.XXX...................] duration: 452.785107ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:08:28 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 452.822082ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [................XX.....X.XXXXXX.X..XXX......................] duration: 1.711262316s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:08:27 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.711352494s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.................X....X...XXX..XX.X..X..X.XX................] duration: 271.331445ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:08:22 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 271.35745ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.................XX.....X.XXXXXX.X..XXX.....................] duration: 1.857894954s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:08:06 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.857927901s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [................X......XXXXXX.X...XX.X.X....................] duration: 235.528233ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:08:03 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 235.557601ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 212.90594ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:08:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 212.936918ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX..................] duration: 485.80892ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:07:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.846592ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..................X....X...XXX..XX.X..X..X.XX...............] duration: 243.414246ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:07:48 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 243.44308ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 207.881953ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:07:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 207.939571ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......................XXXXXX.....XXXX.XXX..................] duration: 450.318821ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:07:41 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 450.358682ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.................X......XXXXXX.X...XX.X.X...................] duration: 353.584948ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:07:39 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 353.622594ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..................XX.....X.XXXXXX.X..XXX....................] duration: 1.663542253s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:07:25 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.663591642s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 195.359288ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:07:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 195.41264ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..................X......XXXXXX.X...XX.X.X..................] duration: 310.600886ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:06:58 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 310.621579ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX.................] duration: 721.049654ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:06:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 721.335078ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...................X....X...XXX..XX.X..X..X.XX..............] duration: 266.831027ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:06:42 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 266.870821ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...................X......XXXXXX.X...XX.X.X.................] duration: 318.509968ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:06:41 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 318.561212ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................XXXXXX.....XXXX.XXX.................] duration: 485.380406ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:06:36 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 485.411699ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...................XX.....X.XXXXXX.X..XXX...................] duration: 1.759976919s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:06:36 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.760007796s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 407.258632ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:06:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 407.291527ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [X.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX................] duration: 606.972003ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:06:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 606.998424ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 281.251303ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:05:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 281.356286ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................XXXXXX.....XXXX.XXX................] duration: 472.068462ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:05:48 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 472.103664ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....................XX.....X.XXXXXX.X..XXX..................] duration: 1.740424656s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:05:45 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.740478014s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [....................X....X...XXX..XX.X..X..X.XX.............] duration: 227.561219ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:05:43 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 227.673659ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [....................X......XXXXXX.X...XX.X.X................] duration: 233.236296ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:05:33 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 233.262901ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX...............] duration: 726.637854ms trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:05:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 726.693668ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....................XX.....X.XXXXXX.X..XXX.................] duration: 1.657119272s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:05:26 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.657149354s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.....................X....X...XXX..XX.X..X..X.XX............] duration: 225.372678ms trace: https://jaeger.dev.storj.io/trace/7c4371cfa8f7834c FAIL (2026-03-18 22:05:06 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 225.405325ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................XXXXXX.....XXXX.XXX...............] duration: 553.643442ms trace: https://jaeger.dev.storj.io/trace/6dceff78c86f0774 FAIL (2026-03-18 22:05:04 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:17.190275520 (298h54m14.787046534s) [upload=334781]: Starting upload Mar 18 21:54:17.190367052 (298h54m14.787138068s) [upload=334781]: Done starting upload Mar 18 21:54:17.190448332 (298h54m14.787219348s) [upload=334781]: Uploading segments... Mar 18 21:54:17.190705184 (298h54m14.787476207s) [upload=334781]: Got next segment. Inline: false Mar 18 21:54:17.190783980 (298h54m14.787555002s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:17.190804859 (298h54m14.787575882s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:25.452304457 (298h54m23.049075487s) [seg_pos={0 0}, upload=334781]: Begin upload segment... Mar 18 21:54:25.452374690 (298h54m23.049145713s) [seg_pos={0 0}, upload=334781]: Done begin upload segment. Mar 18 21:54:25.452463335 (298h54m23.049234366s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.452479705 (298h54m23.049250739s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.453224160 (298h54m23.049995188s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454147610 (298h54m23.050918643s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.454790080 (298h54m23.051561121s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.455678920 (298h54m23.052449952s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456387102 (298h54m23.053158136s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.456987483 (298h54m23.053758515s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.457980795 (298h54m23.054751829s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.459806473 (298h54m23.056577503s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.460305248 (298h54m23.057076281s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.461552822 (298h54m23.058323848s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.463144575 (298h54m23.059915608s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.464837828 (298h54m23.06160886s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.466411504 (298h54m23.063182537s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.469091278 (298h54m23.065862311s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Uploading piece... Mar 18 21:54:25.586976347 (298h54m23.183747361s) [upload=334781]: Next returned nil segment Mar 18 21:54:25.587000505 (298h54m23.183771534s) [upload=334781]: Waiting for error group managing segments... Mar 18 21:54:25.596319798 (298h54m23.193090825s) [noise=true , address=79.127.219.41:4926 , piece_id=AVSCKJ5WDK4OBWWUSLUQNOPZI7BGTTDGT4XJYSLQAJPUHRYY4KVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.596985876 (298h54m23.1937569s) [noise=true , address=45.91.116.9:4264 , piece_id=EE3TXU5KSPHKMNNCI3BHIGFKQNYZQRBDO4YT3TTR5U66YJMFT6XQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598026901 (298h54m23.194797934s) [noise=true , address=145.79.227.26:4656 , piece_id=DDNFTYMUJMFHTZXFHJTFUURSVOJBQX2KVT6UWLBCZ5TIOVBDVOVQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598566162 (298h54m23.195337186s) [noise=true , address=145.79.227.50:4376 , piece_id=BHABIBKA3ZENQUNU2OVBVUJZVWEQRZD435EHA3MEB25KMNBVFQIA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.598781105 (298h54m23.19555213s) [noise=true , address=77.90.31.16:4737 , piece_id=SMQOC6BKSIEIE5FXEYZGOIWRZ6GORC7GCOVEOFZXPQX4FGILVIPA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600260475 (298h54m23.197031509s) [noise=true , address=145.79.227.6:4261 , piece_id=CYOWHNR7X3Y4YE5SFHPNDBOU5QQ6M6DZIRAO3VX56KQBL3OP2FLQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.600344398 (298h54m23.197115411s) [noise=true , address=145.79.227.4:4190 , piece_id=LDID6UGFJ4GWYK635S5OE6334DEULOPVSOLOCLYQCBOFTXRZEKRQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602467676 (298h54m23.19923871s) [noise=true , address=145.79.227.2:4116 , piece_id=KEKK3M6M37G7Y7HXPULI2RSX5CFGUNCM3LILYNQOZ2PASM742ZGQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.602977247 (298h54m23.199748272s) [noise=true , address=145.79.227.48:4307 , piece_id=D6WSRVCL6UO4KQ542EKZWA6IP7FLA3ZUGSBXCSZUMLPFCJ5G5CQA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603520888 (298h54m23.200291919s) [noise=true , address=79.127.219.45:4855 , piece_id=644Z2MAXJ423ZF4JGZ2YLPM6XJRZT7IUJCZRRE7ZJMHB3EW7KB4Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.603907549 (298h54m23.200678573s) [noise=true , address=145.79.227.36:4518 , piece_id=R5PDKAATCPFI3LOOUX6HJ4OP64NQ72N3RAOZS3GYFH5VSWPFU6OQ, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.604438080 (298h54m23.201209113s) [noise=true , address=45.91.116.11:4118 , piece_id=CVKN5WPP2OTRAHYWJKDLCJV3OZ3I6G5CLYXTSEKJDQD7LRTJ55EA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.618924566 (298h54m23.21569559s) [noise=true , address=79.127.219.46:4892 , piece_id=533AWUNWA66VX4AZF7Y3AANCF65GMESWZD3IJWBC5IOKB4QMVTZA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: <nil> Mar 18 21:54:25.619077190 (298h54m23.215848214s) [seg_pos={0 0}, upload=334781]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:25.619546929 (298h54m23.216317961s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.619578476 (298h54m23.21634951s) [noise=true , address=145.79.227.46:4604 , piece_id=DAGTJ24IRMED5FNGQHRYTXEIXYTDVSRW2IT23MC2UC45SES3PE7Q, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620166659 (298h54m23.216937744s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620178720 (298h54m23.216949746s) [noise=true , address=145.79.227.19:4080 , piece_id=ANOMNO3HLII5GPWIT2RDX6P7WOBTVJXMOJM5PAPDMK7ST2NYJWMA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620702987 (298h54m23.217474021s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:25.620718441 (298h54m23.217489474s) [noise=true , address=145.79.227.37:4569 , piece_id=PTE4YT3YJ5M3NDKXKQOUFTGQ3T5FDAUZB74QHJBVH6HQL53QXLUA, seg_pos={0 0}, upload=334781]: Long tail cancellation due to optimal threshold Mar 18 21:54:25.620748329 (298h54m23.21751937s) [seg_pos={0 0}, upload=334781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:25.620912612 (298h54m23.21768364s) [upload=334781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:25.620920854 (298h54m23.217691883s) [upload=334781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:38.136688455 (298h54m35.733459485s) [upload=334781]: Done uploading segments... Mar 18 21:54:38.136713048 (298h54m35.73348408s) [upload=334781]: Upload finished. err: <nil> Duration: 553.695268ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......................XX.....X.XXXXXX.X..XXX................] duration: 1.802056344s trace: https://jaeger.dev.storj.io/trace/27ed2876d5f3e85d FAIL (2026-03-18 22:04:42 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.802086674s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 330.40601ms trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 22:04:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 330.448741ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXX..............] duration: 1.074775062s trace: https://jaeger.dev.storj.io/trace/6cabdd7d47c9cb44 FAIL (2026-03-18 22:04:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.074813238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.....................X......XXXXXX.X...XX.X.X...............] duration: 236.526921ms trace: https://jaeger.dev.storj.io/trace/47004c0815b844e7 FAIL (2026-03-18 22:04:29 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:53:36.090826124 (298h53m33.687597155s) [upload=334767]: Starting upload Mar 18 21:53:36.090977797 (298h53m33.687748822s) [upload=334767]: Done starting upload Mar 18 21:53:36.091078093 (298h53m33.687849119s) [upload=334767]: Uploading segments... Mar 18 21:53:36.091268776 (298h53m33.688040083s) [upload=334767]: Got next segment. Inline: true Mar 18 21:53:36.091279074 (298h53m33.688050103s) [upload=334767]: Waiting for error group managing segments... Mar 18 21:53:36.091372978 (298h53m33.688144001s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:53:36.091379985 (298h53m33.688151008s) [upload=334767]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:53:36.091384265 (298h53m33.688155287s) [upload=334767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:04.718146694 (298h54m2.314917717s) [upload=334767]: Done uploading segments... Mar 18 21:54:04.718169125 (298h54m2.314940155s) [upload=334767]: Upload finished. err: <nil> Duration:* 236.559293ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)