Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [................................................X......XXXXX] duration: 30.002180872s trace: https://jaeger.dev.storj.io/trace/7b1c728d3b51c6cc FAIL (2026-03-18 21:45:31 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:01.575999056 (298h44m59.172770085s) [upload=334616]: Starting upload Mar 18 21:45:01.576112471 (298h44m59.172883505s) [upload=334616]: Done starting upload Mar 18 21:45:01.576293812 (298h44m59.173064822s) [upload=334616]: Uploading segments... Mar 18 21:45:01.576378483 (298h44m59.173149533s) [upload=334616]: Got next segment. Inline: true Mar 18 21:45:01.576403876 (298h44m59.173174906s) [upload=334616]: Waiting for error group managing segments... Mar 18 21:45:01.576508317 (298h44m59.173279344s) [upload=334616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:01.576516634 (298h44m59.173287663s) [upload=334616]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:45:01.576521448 (298h44m59.173292474s) [upload=334616]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:26.144583020 (298h45m23.741354046s) [upload=334616]: Done uploading segments... Mar 18 21:45:26.144616708 (298h45m23.741387735s) [upload=334616]: Upload finished. err: <nil> Duration:* 30.002198853s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXX] duration: 5.000781855s trace: https://jaeger.dev.storj.io/trace/30baa31d7b828883 FAIL (2026-03-18 21:45:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000803149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................XX.....X.XXXX] duration: 30.006576335s trace: https://jaeger.dev.storj.io/trace/1fa8004e24e9d965 FAIL (2026-03-18 21:45:14 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.006590638s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.002729455s trace: https://jaeger.dev.storj.io/trace/322e23d3400fdeb1 FAIL (2026-03-18 21:45:08 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:38.687766729 (298h44m36.284537762s) [upload=334609]: Starting upload Mar 18 21:44:38.687935813 (298h44m36.28470684s) [upload=334609]: Done starting upload Mar 18 21:44:38.688047388 (298h44m36.284818412s) [upload=334609]: Uploading segments... Mar 18 21:44:38.688413967 (298h44m36.285184991s) [upload=334609]: Got next segment. Inline: false Mar 18 21:44:38.688513596 (298h44m36.285284623s) [upload=334609]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:38.688521478 (298h44m36.285292504s) [upload=334609]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:46.853304180 (298h44m44.450075205s) [seg_pos={0 0}, upload=334609]: Begin upload segment... Mar 18 21:44:46.853409197 (298h44m44.450180229s) [noise=true , address=145.79.227.26:4658 , piece_id=YI3Q6WTWRNEHAFUR5WJSUZKPD2H2ABVPXCCOHTMADBBVYIXJ2PAA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.855172779 (298h44m44.451943813s) [seg_pos={0 0}, upload=334609]: Done begin upload segment. Mar 18 21:44:46.855271192 (298h44m44.452042222s) [noise=true , address=77.90.31.10:4522 , piece_id=NGREEEXFBEFVHD2PJSVCGNRHCU2TXAEXE474G4MJISFI3QWACSRQ, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.856033336 (298h44m44.452804368s) [noise=true , address=145.79.227.2:4116 , piece_id=7METYHKRA7XGDJGQTX3LPZOLIVCO66HJ3JZYOJHHMOK6TXUZB7JA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.856620106 (298h44m44.453391139s) [noise=true , address=145.79.227.13:4806 , piece_id=UST75CDBSKEXGK45GRKHU63PA7WPBTW5MSWDXYYJDNNC7MFN2TOA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.857162153 (298h44m44.453933184s) [noise=true , address=145.79.227.31:4802 , piece_id=KCHRKSASK2QLAJM7MVJSEQRFNDO5GJ6GGANSSQHOZKE7CWVJNCXQ, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.857705422 (298h44m44.454476455s) [noise=true , address=45.91.116.4:4972 , piece_id=RVXNJ765IS7TMF6AHMDI6K5MUQXERJWHJY4BMHWU2IUL6Y4IFJGA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.858253532 (298h44m44.45502457s) [noise=true , address=145.79.227.24:4587 , piece_id=SZIPS47CBE7QEF7HFAHD65RVH3SS3ORPV3F66VRJR64YMHPDWG2Q, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.858815898 (298h44m44.45558693s) [noise=true , address=45.91.116.8:4047 , piece_id=PNKRN4MGLMCSGPI5V65KNGRWDNSA4DCCVJNKC6TW2AVPPD6EGTJQ, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.860148075 (298h44m44.456919108s) [noise=true , address=79.127.219.41:4928 , piece_id=75V6EV2MCGYB6SYEKFORFSIMA2NN3MIKEFLZUZSFZETGLVWXUKEA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.863234073 (298h44m44.460005112s) [noise=true , address=145.79.227.15:4879 , piece_id=7I7VSVU2UCKRDCDAVQGE2G65TYFT7ZKBCQE7EDNKOIG3MYOXBYEA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.865670568 (298h44m44.462441602s) [noise=true , address=79.127.219.45:4854 , piece_id=UTWY5Y62HWM2U6BGTW4XMSOFUQTBROR3MWXWTCM3WIBYED44QSMQ, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.867866081 (298h44m44.464637115s) [noise=true , address=145.79.227.42:4127 , piece_id=5P2FV2555HV7VOJGID6SUYHK2F67LVKOHRONZJUKRUGYIPN2D5MA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.869910241 (298h44m44.466681277s) [noise=true , address=145.79.227.39:4017 , piece_id=K62BOFYTWUU2AAPJLD7PUMX32XNN7H3U4QFI4XTHO2ESNI75IBCA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.871730562 (298h44m44.468501595s) [noise=true , address=79.127.219.38:4818 , piece_id=NIVLENEZFPRMMFP7ZU7PTQKXISL5IZFH5LCJOHW3LL7RVCW4OTDA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.873529778 (298h44m44.470300813s) [noise=true , address=145.79.227.32:4838 , piece_id=5LWOJSYRB76RDONKSSXJ2PV7VQX57RWOK3XB6X2A7XDAC2I4KPLA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.873657270 (298h44m44.470428305s) [noise=true , address=145.79.227.47:4266 , piece_id=JN3NGNDLBL7ELOXDAU2LXTSNQCLQOCXUK5M2GHLWWNFYQUJ2PCQA, seg_pos={0 0}, upload=334609]: Uploading piece... Mar 18 21:44:46.948323168 (298h44m44.545094199s) [upload=334609]: Next returned nil segment Mar 18 21:44:46.948342635 (298h44m44.545113668s) [upload=334609]: Waiting for error group managing segments... Mar 18 21:44:46.953716547 (298h44m44.550487576s) [noise=true , address=45.91.116.8:4047 , piece_id=PNKRN4MGLMCSGPI5V65KNGRWDNSA4DCCVJNKC6TW2AVPPD6EGTJQ, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.953930857 (298h44m44.550701884s) [noise=true , address=45.91.116.4:4972 , piece_id=RVXNJ765IS7TMF6AHMDI6K5MUQXERJWHJY4BMHWU2IUL6Y4IFJGA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.954507138 (298h44m44.55127817s) [noise=true , address=77.90.31.10:4522 , piece_id=NGREEEXFBEFVHD2PJSVCGNRHCU2TXAEXE474G4MJISFI3QWACSRQ, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.956719642 (298h44m44.553490669s) [noise=true , address=145.79.227.15:4879 , piece_id=7I7VSVU2UCKRDCDAVQGE2G65TYFT7ZKBCQE7EDNKOIG3MYOXBYEA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.957651163 (298h44m44.554422195s) [noise=true , address=79.127.219.45:4854 , piece_id=UTWY5Y62HWM2U6BGTW4XMSOFUQTBROR3MWXWTCM3WIBYED44QSMQ, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.958615658 (298h44m44.55538667s) [noise=true , address=145.79.227.26:4658 , piece_id=YI3Q6WTWRNEHAFUR5WJSUZKPD2H2ABVPXCCOHTMADBBVYIXJ2PAA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.958642568 (298h44m44.555413597s) [noise=true , address=145.79.227.24:4587 , piece_id=SZIPS47CBE7QEF7HFAHD65RVH3SS3ORPV3F66VRJR64YMHPDWG2Q, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.959280712 (298h44m44.556051737s) [noise=true , address=145.79.227.42:4127 , piece_id=5P2FV2555HV7VOJGID6SUYHK2F67LVKOHRONZJUKRUGYIPN2D5MA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.960286406 (298h44m44.55705743s) [noise=true , address=145.79.227.13:4806 , piece_id=UST75CDBSKEXGK45GRKHU63PA7WPBTW5MSWDXYYJDNNC7MFN2TOA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.960792202 (298h44m44.557563225s) [noise=true , address=145.79.227.2:4116 , piece_id=7METYHKRA7XGDJGQTX3LPZOLIVCO66HJ3JZYOJHHMOK6TXUZB7JA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.961166710 (298h44m44.557937736s) [noise=true , address=145.79.227.31:4802 , piece_id=KCHRKSASK2QLAJM7MVJSEQRFNDO5GJ6GGANSSQHOZKE7CWVJNCXQ, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.961745747 (298h44m44.558516774s) [noise=true , address=145.79.227.32:4838 , piece_id=5LWOJSYRB76RDONKSSXJ2PV7VQX57RWOK3XB6X2A7XDAC2I4KPLA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.966954257 (298h44m44.56372528s) [noise=true , address=79.127.219.38:4818 , piece_id=NIVLENEZFPRMMFP7ZU7PTQKXISL5IZFH5LCJOHW3LL7RVCW4OTDA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: <nil> Mar 18 21:44:46.966976103 (298h44m44.563747116s) [seg_pos={0 0}, upload=334609]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:46.967231062 (298h44m44.56400209s) [noise=true , address=145.79.227.47:4266 , piece_id=JN3NGNDLBL7ELOXDAU2LXTSNQCLQOCXUK5M2GHLWWNFYQUJ2PCQA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:46.967239412 (298h44m44.564010436s) [noise=true , address=145.79.227.47:4266 , piece_id=JN3NGNDLBL7ELOXDAU2LXTSNQCLQOCXUK5M2GHLWWNFYQUJ2PCQA, seg_pos={0 0}, upload=334609]: Long tail cancellation due to optimal threshold Mar 18 21:44:46.967665480 (298h44m44.564436498s) [noise=true , address=79.127.219.41:4928 , piece_id=75V6EV2MCGYB6SYEKFORFSIMA2NN3MIKEFLZUZSFZETGLVWXUKEA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pvLo4F5ieDtepopg8Wi8jw6PKNeDCDvtH5iVNQXBUQwSXYxuD): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:46.967678412 (298h44m44.564449428s) [noise=true , address=79.127.219.41:4928 , piece_id=75V6EV2MCGYB6SYEKFORFSIMA2NN3MIKEFLZUZSFZETGLVWXUKEA, seg_pos={0 0}, upload=334609]: Long tail cancellation due to optimal threshold Mar 18 21:44:46.967825484 (298h44m44.56459651s) [noise=true , address=145.79.227.39:4017 , piece_id=K62BOFYTWUU2AAPJLD7PUMX32XNN7H3U4QFI4XTHO2ESNI75IBCA, seg_pos={0 0}, upload=334609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:46.967832432 (298h44m44.564603455s) [noise=true , address=145.79.227.39:4017 , piece_id=K62BOFYTWUU2AAPJLD7PUMX32XNN7H3U4QFI4XTHO2ESNI75IBCA, seg_pos={0 0}, upload=334609]: Long tail cancellation due to optimal threshold Mar 18 21:44:46.967850015 (298h44m44.564621028s) [seg_pos={0 0}, upload=334609]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:46.967927195 (298h44m44.564698218s) [upload=334609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:46.967933482 (298h44m44.564704505s) [upload=334609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:07.465901400 (298h45m5.062672436s) [upload=334609]: Done uploading segments... Mar 18 21:45:07.465928035 (298h45m5.062699062s) [upload=334609]: Upload finished. err: <nil> Duration: 30.002755248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X....X...XXX..X] duration: 30.001231076s trace: https://jaeger.dev.storj.io/trace/5e0333507174f11b FAIL (2026-03-18 21:45:05 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001256138s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X......XXXX] duration: 30.001942696s trace: https://jaeger.dev.storj.io/trace/13ca68dca9aa47ed FAIL (2026-03-18 21:45:01 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:31.519656014 (298h44m29.116427038s) [upload=334606]: Starting upload Mar 18 21:44:31.519709243 (298h44m29.116480276s) [upload=334606]: Done starting upload Mar 18 21:44:31.519772107 (298h44m29.116543123s) [upload=334606]: Uploading segments... Mar 18 21:44:31.519811691 (298h44m29.116582723s) [upload=334606]: Got next segment. Inline: true Mar 18 21:44:31.519816642 (298h44m29.116587665s) [upload=334606]: Waiting for error group managing segments... Mar 18 21:44:31.519857234 (298h44m29.116628258s) [upload=334606]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:31.519863787 (298h44m29.116634817s) [upload=334606]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:31.519867315 (298h44m29.116638339s) [upload=334606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:01.460804879 (298h44m59.057575915s) [upload=334606]: Done uploading segments... Mar 18 21:45:01.460862448 (298h44m59.057633483s) [upload=334606]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001966538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 29.039879221s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:45:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 29.039952443s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX.....X.XXX] duration: 30.006981897s trace: https://jaeger.dev.storj.io/trace/703aa358f4d3eca0 FAIL (2026-03-18 21:44:44 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00708591s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXX] duration: 5.001644565s trace: https://jaeger.dev.storj.io/trace/180373c9302b5643 FAIL (2026-03-18 21:44:42 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001679334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 49.15005685s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:44:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 49.150085599s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X....X...XXX..] duration: 29.011782791s trace: https://jaeger.dev.storj.io/trace/729bbf15f1a2e3d FAIL (2026-03-18 21:44:28 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 29.011809943s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXX] duration: 5.001294451s trace: https://jaeger.dev.storj.io/trace/3b037cf2f912fad FAIL (2026-03-18 21:44:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001321947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.002401169s trace: https://jaeger.dev.storj.io/trace/96c02eb15ce515 FAIL (2026-03-18 21:43:58 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:43:28.258084883 (298h43m25.854855907s) [upload=334589]: Starting upload Mar 18 21:43:28.258200792 (298h43m25.854971807s) [upload=334589]: Done starting upload Mar 18 21:43:28.258244859 (298h43m25.855015874s) [upload=334589]: Uploading segments... Mar 18 21:43:28.258765520 (298h43m25.855536543s) [upload=334589]: Got next segment. Inline: false Mar 18 21:43:28.258807483 (298h43m25.855578506s) [upload=334589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:43:28.258813703 (298h43m25.855584726s) [upload=334589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:43:44.524662780 (298h43m42.121433805s) [seg_pos={0 0}, upload=334589]: Begin upload segment... Mar 18 21:43:44.524731425 (298h43m42.121502447s) [seg_pos={0 0}, upload=334589]: Done begin upload segment. Mar 18 21:43:44.524762851 (298h43m42.121533874s) [noise=true , address=145.79.227.9:4369 , piece_id=FSTQI4TFZWURSPQ6CGWTZZSUTNE2QK3ST42E3O7BV4VM33BHCY2Q, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.525904092 (298h43m42.122675125s) [noise=true , address=145.79.227.29:4944 , piece_id=K46XQYTSPFMDGMVPX4BYSWAYMQCSCN3DNARME7Q3XXMU5KNNUW2Q, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.526023915 (298h43m42.122794947s) [noise=true , address=145.79.227.45:4230 , piece_id=K5XD3YYP7E4ZMYZHL62AYWVARKPFFMKHP4L6M2J6UWL2DG5BALHA, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.526490628 (298h43m42.123261657s) [noise=true , address=45.91.116.13:4154 , piece_id=YCGYSCSV4BHXZ6QAMD4WP4XJKSNY5CBX3OWFASLI4WEB7CIMZ2KA, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.526577800 (298h43m42.123348833s) [noise=true , address=145.79.227.33:4874 , piece_id=ND2G4LTYJI4XASRVYZPWNT6XUA64ZWZIU3MJI3UUCRFMEPCKEHAQ, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.526889875 (298h43m42.123660898s) [noise=true , address=145.79.227.42:4122 , piece_id=ITAZDO3VI2BJCGR32CYB6O6CKOEYWGGKSUHBWHNZFWNN2FQFEGXA, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.527271442 (298h43m42.124042468s) [noise=true , address=145.79.227.46:4601 , piece_id=4X4FOT5R7YMPGRQ2DPJSWUBYMMZ3UZPVVAVH2E2B3QSJ7J5UH2UQ, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.527317133 (298h43m42.124088161s) [noise=true , address=145.79.227.7:4299 , piece_id=OWNISTHVCXH7LW5TSP274FT736QFJULVSHLIDPWMAN267L5ON7SQ, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.527754316 (298h43m42.124525343s) [noise=true , address=145.79.227.17:4974 , piece_id=2ZVJ4QRFST5ZOHODBBW2FPXZER4JWQE2RANDDXFRQU3A73TOC2VQ, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.534731304 (298h43m42.131502329s) [noise=true , address=145.79.227.10:4405 , piece_id=MLM3VBAW5FQTGYSIBSXS52GMLDO577266Y52OLXY4ORB7NPW542A, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.534859355 (298h43m42.131630378s) [noise=true , address=145.79.227.44:4196 , piece_id=3UULZ7FDSVSBQAV2YBKST2LYPR5GAF5J2GXUKJSYIFYHMY2NXFAA, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.536465860 (298h43m42.133236889s) [noise=true , address=145.79.227.47:4269 , piece_id=LO24JEN6LKWMVO6FE5PVVDDLI6IDWVTWJAHO4OJ75VMGKD3D4G3A, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.537440676 (298h43m42.134211707s) [noise=true , address=145.79.227.49:4338 , piece_id=OIQD2TL72DG3TVO5UW5P5FWLCBGVA3JFARIWI2XX4SLB2ATNJQMQ, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.539495854 (298h43m42.136266887s) [noise=true , address=145.79.227.28:4729 , piece_id=UKTWPLZT5FVPN4OURPXPABVMKH73XQVF47O3MAJOPV5GC2WV2GJA, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.542450471 (298h43m42.139221502s) [noise=true , address=45.91.116.11:4117 , piece_id=3EFCS4NE2EHZNIMJ42TJYHD3QJ3VAEQPGAYWHZTXAEMUSS66SO6Q, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.546476579 (298h43m42.143247609s) [noise=true , address=145.79.227.26:4656 , piece_id=JUCJV3VDH2SSV6OHXMY6HOWANZRTFICSZH4WURWQNAQ7WFNKK7RQ, seg_pos={0 0}, upload=334589]: Uploading piece... Mar 18 21:43:44.614307688 (298h43m42.211078711s) [upload=334589]: Next returned nil segment Mar 18 21:43:44.614325459 (298h43m42.211096493s) [upload=334589]: Waiting for error group managing segments... Mar 18 21:43:44.624800746 (298h43m42.221571772s) [noise=true , address=145.79.227.26:4656 , piece_id=JUCJV3VDH2SSV6OHXMY6HOWANZRTFICSZH4WURWQNAQ7WFNKK7RQ, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.625627807 (298h43m42.222398821s) [noise=true , address=45.91.116.13:4154 , piece_id=YCGYSCSV4BHXZ6QAMD4WP4XJKSNY5CBX3OWFASLI4WEB7CIMZ2KA, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.626138140 (298h43m42.222909167s) [noise=true , address=145.79.227.49:4338 , piece_id=OIQD2TL72DG3TVO5UW5P5FWLCBGVA3JFARIWI2XX4SLB2ATNJQMQ, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.626792293 (298h43m42.223563319s) [noise=true , address=45.91.116.11:4117 , piece_id=3EFCS4NE2EHZNIMJ42TJYHD3QJ3VAEQPGAYWHZTXAEMUSS66SO6Q, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.627409347 (298h43m42.22418037s) [noise=true , address=145.79.227.42:4122 , piece_id=ITAZDO3VI2BJCGR32CYB6O6CKOEYWGGKSUHBWHNZFWNN2FQFEGXA, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.627590671 (298h43m42.224361697s) [noise=true , address=145.79.227.17:4974 , piece_id=2ZVJ4QRFST5ZOHODBBW2FPXZER4JWQE2RANDDXFRQU3A73TOC2VQ, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.628436374 (298h43m42.225207456s) [noise=true , address=145.79.227.9:4369 , piece_id=FSTQI4TFZWURSPQ6CGWTZZSUTNE2QK3ST42E3O7BV4VM33BHCY2Q, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.628554100 (298h43m42.225325135s) [noise=true , address=145.79.227.10:4405 , piece_id=MLM3VBAW5FQTGYSIBSXS52GMLDO577266Y52OLXY4ORB7NPW542A, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.629404730 (298h43m42.226175758s) [noise=true , address=145.79.227.45:4230 , piece_id=K5XD3YYP7E4ZMYZHL62AYWVARKPFFMKHP4L6M2J6UWL2DG5BALHA, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.629565933 (298h43m42.226336959s) [noise=true , address=145.79.227.7:4299 , piece_id=OWNISTHVCXH7LW5TSP274FT736QFJULVSHLIDPWMAN267L5ON7SQ, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.630571503 (298h43m42.227342515s) [noise=true , address=145.79.227.46:4601 , piece_id=4X4FOT5R7YMPGRQ2DPJSWUBYMMZ3UZPVVAVH2E2B3QSJ7J5UH2UQ, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.631168525 (298h43m42.22793954s) [noise=true , address=145.79.227.29:4944 , piece_id=K46XQYTSPFMDGMVPX4BYSWAYMQCSCN3DNARME7Q3XXMU5KNNUW2Q, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.643781680 (298h43m42.24055271s) [noise=true , address=145.79.227.33:4874 , piece_id=ND2G4LTYJI4XASRVYZPWNT6XUA64ZWZIU3MJI3UUCRFMEPCKEHAQ, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.643810376 (298h43m42.240581399s) [seg_pos={0 0}, upload=334589]: Satisfied cohort requirements after 13 pieces Mar 18 21:43:44.644020628 (298h43m42.240791657s) [noise=true , address=145.79.227.28:4729 , piece_id=UKTWPLZT5FVPN4OURPXPABVMKH73XQVF47O3MAJOPV5GC2WV2GJA, seg_pos={0 0}, upload=334589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L): protocol: expected piece hash; context canceled; EOF Mar 18 21:43:44.644028681 (298h43m42.240799705s) [noise=true , address=145.79.227.28:4729 , piece_id=UKTWPLZT5FVPN4OURPXPABVMKH73XQVF47O3MAJOPV5GC2WV2GJA, seg_pos={0 0}, upload=334589]: Long tail cancellation due to optimal threshold Mar 18 21:43:44.644189775 (298h43m42.240960805s) [noise=true , address=145.79.227.44:4196 , piece_id=3UULZ7FDSVSBQAV2YBKST2LYPR5GAF5J2GXUKJSYIFYHMY2NXFAA, seg_pos={0 0}, upload=334589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Mar 18 21:43:44.644196345 (298h43m42.240967369s) [noise=true , address=145.79.227.44:4196 , piece_id=3UULZ7FDSVSBQAV2YBKST2LYPR5GAF5J2GXUKJSYIFYHMY2NXFAA, seg_pos={0 0}, upload=334589]: Long tail cancellation due to optimal threshold Mar 18 21:43:44.644607697 (298h43m42.241378711s) [noise=true , address=145.79.227.47:4269 , piece_id=LO24JEN6LKWMVO6FE5PVVDDLI6IDWVTWJAHO4OJ75VMGKD3D4G3A, seg_pos={0 0}, upload=334589]: Done uploading piece. err: <nil> Mar 18 21:43:44.644623706 (298h43m42.241394723s) [seg_pos={0 0}, upload=334589]: Satisfied cohort requirements after 14 pieces Mar 18 21:43:44.644652910 (298h43m42.241423922s) [seg_pos={0 0}, upload=334589]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Mar 18 21:43:44.644740810 (298h43m42.241511824s) [upload=334589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:43:44.644747552 (298h43m42.241518576s) [upload=334589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:43:53.020673790 (298h43m50.617444822s) [upload=334589]: Done uploading segments... Mar 18 21:43:53.020700730 (298h43m50.617471762s) [upload=334589]: Upload finished. err: <nil> Duration: 30.002427023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX.....X.XX] duration: 30.006554488s trace: https://jaeger.dev.storj.io/trace/276e3c3c00e5a8bd FAIL (2026-03-18 21:43:53 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.006587476s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X......XXX] duration: 30.001007996s trace: https://jaeger.dev.storj.io/trace/2b5504fc774734db FAIL (2026-03-18 21:43:48 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:43:18.595881357 (298h43m16.19265239s) [upload=334585]: Starting upload Mar 18 21:43:18.595983408 (298h43m16.192754432s) [upload=334585]: Done starting upload Mar 18 21:43:18.596043424 (298h43m16.192814447s) [upload=334585]: Uploading segments... Mar 18 21:43:18.596138670 (298h43m16.192909703s) [upload=334585]: Got next segment. Inline: true Mar 18 21:43:18.596158809 (298h43m16.192929833s) [upload=334585]: Waiting for error group managing segments... Mar 18 21:43:18.596267549 (298h43m16.193038573s) [upload=334585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:43:18.596275448 (298h43m16.193046476s) [upload=334585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:43:18.596281549 (298h43m16.193052562s) [upload=334585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:43:31.057165231 (298h43m28.653936254s) [upload=334585]: Done uploading segments... Mar 18 21:43:31.057185921 (298h43m28.653956945s) [upload=334585]: Upload finished. err: <nil> Duration:* 30.0010205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 45.03725604s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:43:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 45.037275608s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X....X...XXX.] duration: 20.929682467s trace: https://jaeger.dev.storj.io/trace/729bbf15f1a2e3d FAIL (2026-03-18 21:43:21 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 20.929701079s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X......XX] duration: 30.001361801s trace: https://jaeger.dev.storj.io/trace/6de3d0ca05c37a3b FAIL (2026-03-18 21:43:18 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:48.416425885 (298h42m46.01319692s) [upload=334577]: Starting upload Mar 18 21:42:48.416509067 (298h42m46.013280091s) [upload=334577]: Done starting upload Mar 18 21:42:48.416616529 (298h42m46.013387556s) [upload=334577]: Uploading segments... Mar 18 21:42:48.416671672 (298h42m46.013442716s) [upload=334577]: Got next segment. Inline: true Mar 18 21:42:48.416678563 (298h42m46.013449595s) [upload=334577]: Waiting for error group managing segments... Mar 18 21:42:48.416746426 (298h42m46.013517458s) [upload=334577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:48.416753058 (298h42m46.013524092s) [upload=334577]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:48.416757788 (298h42m46.013528813s) [upload=334577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:43:00.948376458 (298h42m58.545147585s) [upload=334577]: Done uploading segments... Mar 18 21:43:00.948405226 (298h42m58.54517626s) [upload=334577]: Upload finished. err: <nil> Duration:* 30.001383558s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXX] duration: 5.001157988s trace: https://jaeger.dev.storj.io/trace/12de4e66e2552a9b FAIL (2026-03-18 21:43:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001190413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XX.....X.X] duration: 30.005906246s trace: https://jaeger.dev.storj.io/trace/4adab4a1073f731 FAIL (2026-03-18 21:43:08 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005940771s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [................................................X....X...XXX] duration: 30.002029456s trace: https://jaeger.dev.storj.io/trace/729bbf15f1a2e3d FAIL (2026-03-18 21:43:00 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.002055516s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.02352605s trace: https://jaeger.dev.storj.io/trace/68fb9576f5f9fe22 FAIL (2026-03-18 21:42:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:20.970718341 (298h42m18.567489365s) [upload=334570]: Starting upload Mar 18 21:42:20.970814337 (298h42m18.567585359s) [upload=334570]: Done starting upload Mar 18 21:42:20.971198823 (298h42m18.567969839s) [upload=334570]: Uploading segments... Mar 18 21:42:20.972062438 (298h42m18.568833476s) [upload=334570]: Got next segment. Inline: false Mar 18 21:42:20.972247623 (298h42m18.56901864s) [upload=334570]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:20.972258768 (298h42m18.5690298s) [upload=334570]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:29.061589976 (298h42m26.658361004s) [seg_pos={0 0}, upload=334570]: Begin upload segment... Mar 18 21:42:29.061672816 (298h42m26.65844384s) [seg_pos={0 0}, upload=334570]: Done begin upload segment. Mar 18 21:42:29.061731049 (298h42m26.658502073s) [noise=true , address=145.79.227.42:4125 , piece_id=5EVX5MUIITUDGNTQTYXTV4FM6SRYIWAZCU6XCLHUQ6YYC3PBCWDA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.063096235 (298h42m26.659867267s) [noise=true , address=145.79.227.15:4877 , piece_id=QZJ7NLAN4X2CFSPSDCZ3CFQPGW2GYO7WQRDIIBQO4Z3J7R4TE3JA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.064028368 (298h42m26.660799401s) [noise=true , address=77.90.31.19:4841 , piece_id=U66JNTMMQ3WHANYMEMA4NKBZPQOONSIDE2G6E6HUL7XOBTDTBXHQ, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.064498733 (298h42m26.661269802s) [noise=true , address=77.90.31.14:4661 , piece_id=2WNQQT5VNYJ2OVAG6VCCFZAMUXKX6LCJ46ROHV7JWSFHML4P3G4A, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.064942515 (298h42m26.661713552s) [noise=true , address=145.79.227.34:4909 , piece_id=BWK34XHW74MOIWUQ3SD745WVZ7XAWOSYRKTTB34XGHX7DUPOGIPQ, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.065643983 (298h42m26.662415012s) [noise=true , address=145.79.227.24:4587 , piece_id=ZE33ZRF46CVOFCWP4LVMKH36XWF5JNPTKL4Y4YRPUDTAYR4GSW7A, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.065939808 (298h42m26.662710841s) [noise=true , address=45.91.116.11:4152 , piece_id=6AL6XUC3PVHJBUJPB72XUKEEB3S7EV3DKWALPCREK6SCOOUOB7KA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.066594909 (298h42m26.663365939s) [noise=true , address=145.79.227.20:4443 , piece_id=TTVREHQMR7ZVMLLJX2W4VEQWOYXODWPPUEXARLAUAAZWDTR7OIHA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.066843551 (298h42m26.663614581s) [noise=true , address=145.79.227.52:4445 , piece_id=6F3GNKO2GZ3E7ENY7Y32QIZXX2AF5OFGCE6AHBOHKI7ZG3C7XLSA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.067217515 (298h42m26.663988549s) [noise=true , address=145.79.227.40:4050 , piece_id=V7Q5PWJEGQM2ZJ5CJZI7V4446GYFIS2YPJEKCQLC3GEB2PHMPNGQ, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.067395972 (298h42m26.664166986s) [noise=true , address=145.79.227.26:4656 , piece_id=SGRRLW5TTAL6AAX3OZEML3DCOJISQXF4EKQNUAEJIGT2AS2SSB4Q, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.067852641 (298h42m26.66462367s) [noise=true , address=45.91.116.8:4079 , piece_id=W3FDYLJ4IG7HUNQWUWN5QKIUQR3SVUZ7YWQ6CUL6C4WHYP2HWZ4A, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.067987672 (298h42m26.664758705s) [noise=true , address=45.91.116.5:4301 , piece_id=S52KNU6OULYMVLBGFFYROY6PUDQVNMF5JK5QIKBEFTAGIARBANNA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.068467821 (298h42m26.665238844s) [noise=true , address=145.79.227.23:4548 , piece_id=PWRUZQNUGOLR5O34RFW3XPHNGFK4DWKH23OATNYORILIUS762VBA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.068497565 (298h42m26.665268582s) [noise=true , address=79.127.219.46:4890 , piece_id=7DCO35FKPAEQEQBYBOOURAZT2EDTPLRE3QDJXCGNZIUF73JSGPEA, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.068963359 (298h42m26.665734385s) [noise=true , address=145.79.227.7:4298 , piece_id=FEWRGXFHCNGDFXBW6ZZZJRMUVFDJMJJZ6MWCM37F5X3TG4GWUDPQ, seg_pos={0 0}, upload=334570]: Uploading piece... Mar 18 21:42:29.137668218 (298h42m26.734439231s) [upload=334570]: Next returned nil segment Mar 18 21:42:29.137682689 (298h42m26.734453713s) [upload=334570]: Waiting for error group managing segments... Mar 18 21:42:29.148695261 (298h42m26.74546629s) [noise=true , address=145.79.227.15:4877 , piece_id=QZJ7NLAN4X2CFSPSDCZ3CFQPGW2GYO7WQRDIIBQO4Z3J7R4TE3JA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.149134351 (298h42m26.74590538s) [noise=true , address=45.91.116.11:4152 , piece_id=6AL6XUC3PVHJBUJPB72XUKEEB3S7EV3DKWALPCREK6SCOOUOB7KA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.149183613 (298h42m26.745954641s) [noise=true , address=45.91.116.5:4301 , piece_id=S52KNU6OULYMVLBGFFYROY6PUDQVNMF5JK5QIKBEFTAGIARBANNA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.149758282 (298h42m26.746529316s) [noise=true , address=45.91.116.8:4079 , piece_id=W3FDYLJ4IG7HUNQWUWN5QKIUQR3SVUZ7YWQ6CUL6C4WHYP2HWZ4A, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.149853867 (298h42m26.74662488s) [noise=true , address=145.79.227.40:4050 , piece_id=V7Q5PWJEGQM2ZJ5CJZI7V4446GYFIS2YPJEKCQLC3GEB2PHMPNGQ, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.150382185 (298h42m26.747153209s) [noise=true , address=79.127.219.46:4890 , piece_id=7DCO35FKPAEQEQBYBOOURAZT2EDTPLRE3QDJXCGNZIUF73JSGPEA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.150788588 (298h42m26.74755961s) [noise=true , address=145.79.227.26:4656 , piece_id=SGRRLW5TTAL6AAX3OZEML3DCOJISQXF4EKQNUAEJIGT2AS2SSB4Q, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.150891401 (298h42m26.747662425s) [noise=true , address=145.79.227.7:4298 , piece_id=FEWRGXFHCNGDFXBW6ZZZJRMUVFDJMJJZ6MWCM37F5X3TG4GWUDPQ, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.151452609 (298h42m26.748223632s) [noise=true , address=77.90.31.19:4841 , piece_id=U66JNTMMQ3WHANYMEMA4NKBZPQOONSIDE2G6E6HUL7XOBTDTBXHQ, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.152358513 (298h42m26.749129536s) [noise=true , address=145.79.227.52:4445 , piece_id=6F3GNKO2GZ3E7ENY7Y32QIZXX2AF5OFGCE6AHBOHKI7ZG3C7XLSA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.152943736 (298h42m26.749714758s) [noise=true , address=145.79.227.42:4125 , piece_id=5EVX5MUIITUDGNTQTYXTV4FM6SRYIWAZCU6XCLHUQ6YYC3PBCWDA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.153487333 (298h42m26.750258367s) [noise=true , address=77.90.31.14:4661 , piece_id=2WNQQT5VNYJ2OVAG6VCCFZAMUXKX6LCJ46ROHV7JWSFHML4P3G4A, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.153657633 (298h42m26.750428642s) [noise=true , address=145.79.227.23:4548 , piece_id=PWRUZQNUGOLR5O34RFW3XPHNGFK4DWKH23OATNYORILIUS762VBA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: <nil> Mar 18 21:42:29.153673043 (298h42m26.750444075s) [seg_pos={0 0}, upload=334570]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:29.153895552 (298h42m26.750666586s) [noise=true , address=145.79.227.24:4587 , piece_id=ZE33ZRF46CVOFCWP4LVMKH36XWF5JNPTKL4Y4YRPUDTAYR4GSW7A, seg_pos={0 0}, upload=334570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:29.153906082 (298h42m26.750677119s) [noise=true , address=145.79.227.24:4587 , piece_id=ZE33ZRF46CVOFCWP4LVMKH36XWF5JNPTKL4Y4YRPUDTAYR4GSW7A, seg_pos={0 0}, upload=334570]: Long tail cancellation due to optimal threshold Mar 18 21:42:29.153979258 (298h42m26.75075028s) [noise=true , address=145.79.227.20:4443 , piece_id=TTVREHQMR7ZVMLLJX2W4VEQWOYXODWPPUEXARLAUAAZWDTR7OIHA, seg_pos={0 0}, upload=334570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rTVUAPUj4PUy8qHuutnB2WYCVpJdiVwfutA6UK4TZoPzvQEUv): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:29.153989064 (298h42m26.750760092s) [noise=true , address=145.79.227.20:4443 , piece_id=TTVREHQMR7ZVMLLJX2W4VEQWOYXODWPPUEXARLAUAAZWDTR7OIHA, seg_pos={0 0}, upload=334570]: Long tail cancellation due to optimal threshold Mar 18 21:42:29.154170062 (298h42m26.750941102s) [noise=true , address=145.79.227.34:4909 , piece_id=BWK34XHW74MOIWUQ3SD745WVZ7XAWOSYRKTTB34XGHX7DUPOGIPQ, seg_pos={0 0}, upload=334570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:29.154178141 (298h42m26.750949172s) [noise=true , address=145.79.227.34:4909 , piece_id=BWK34XHW74MOIWUQ3SD745WVZ7XAWOSYRKTTB34XGHX7DUPOGIPQ, seg_pos={0 0}, upload=334570]: Long tail cancellation due to optimal threshold Mar 18 21:42:29.154198614 (298h42m26.750969627s) [seg_pos={0 0}, upload=334570]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:29.154285944 (298h42m26.751056955s) [upload=334570]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:29.154291316 (298h42m26.751062346s) [upload=334570]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:50.951788059 (298h42m48.548559085s) [upload=334570]: Done uploading segments... Mar 18 21:42:50.951932533 (298h42m48.548703565s) [upload=334570]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.023550983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002102579s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:42:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 1m0.002125758s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 30.001503801s trace: https://jaeger.dev.storj.io/trace/7d899fda4853140a FAIL (2026-03-18 21:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:57.077587388 (298h41m54.674358401s) [upload=334562]: Starting upload Mar 18 21:41:57.077727663 (298h41m54.674498687s) [upload=334562]: Done starting upload Mar 18 21:41:57.077778987 (298h41m54.67455s) [upload=334562]: Uploading segments... Mar 18 21:41:57.077946793 (298h41m54.674717819s) [upload=334562]: Got next segment. Inline: true Mar 18 21:41:57.077954182 (298h41m54.674725212s) [upload=334562]: Waiting for error group managing segments... Mar 18 21:41:57.078195470 (298h41m54.674966508s) [upload=334562]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:57.078207388 (298h41m54.674978422s) [upload=334562]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:57.078212471 (298h41m54.674983508s) [upload=334562]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:27.048512235 (298h42m24.6452833s) [upload=334562]: Done uploading segments... Mar 18 21:42:27.048564495 (298h42m24.645335529s) [upload=334562]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001520326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XX.....X.] duration: 26.511075215s trace: https://jaeger.dev.storj.io/trace/3f23889572a83afd FAIL (2026-03-18 21:42:23 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.511104032s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXX] duration: 5.000989901s trace: https://jaeger.dev.storj.io/trace/52b41f7de41e8961 FAIL (2026-03-18 21:42:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001011485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.003335143s trace: https://jaeger.dev.storj.io/trace/ec6f2e34311687a FAIL (2026-03-18 21:42:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:37.406754651 (298h41m35.003525674s) [upload=334557]: Starting upload Mar 18 21:41:37.406842992 (298h41m35.003614008s) [upload=334557]: Done starting upload Mar 18 21:41:37.407268182 (298h41m35.004039195s) [upload=334557]: Uploading segments... Mar 18 21:41:37.407347700 (298h41m35.004118725s) [upload=334557]: Got next segment. Inline: false Mar 18 21:41:37.407385125 (298h41m35.004156148s) [upload=334557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:37.407390032 (298h41m35.004161055s) [upload=334557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:41:45.546807947 (298h41m43.143578961s) [seg_pos={0 0}, upload=334557]: Begin upload segment... Mar 18 21:41:45.546881249 (298h41m43.143652273s) [seg_pos={0 0}, upload=334557]: Done begin upload segment. Mar 18 21:41:45.546946135 (298h41m43.143717179s) [noise=true , address=145.79.227.52:4448 , piece_id=MUXR27EU23SUFKVE5BRZPVLJZIZGNTTBOLANCCQ4FDJZZ2QJ55FQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.547169748 (298h41m43.143940781s) [noise=true , address=145.79.227.26:4656 , piece_id=RU2GPQTGMSTCDP2SY4LFWANHCKPI2OIQ2DPHQ6K2N77K72SIH4HQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.547859019 (298h41m43.144630045s) [noise=true , address=145.79.227.4:4190 , piece_id=MD5I5KWECLFX6WRJZZOZ7C5XGNNLJFCY7JN23ETN4ZZNBIYLRTXA, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.548076541 (298h41m43.144847576s) [noise=true , address=145.79.227.6:4263 , piece_id=IOYKTKNT3B65RLSEEQ7EABL4OZV5JNAPVE3ATOWVOT67AVVIZ3KQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.548456056 (298h41m43.145227083s) [noise=true , address=145.79.227.31:4801 , piece_id=YDM3VXUVYWH6J2XNGQB6SWSQHV3XYJBNSQCHXINCTNGVDOZ6EUPQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.548583861 (298h41m43.145354897s) [noise=true , address=145.79.227.13:4807 , piece_id=64GOGTQX6HIMS4FEQ6EGDOK7GUUJHVFKJM7VODTFIMS6GRIGR73A, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.549093611 (298h41m43.145864645s) [noise=true , address=145.79.227.18:4011 , piece_id=PCVRQVDFHFHCD5FYQ54OCIBEIBWATMMFH6MRLIO2KXJ6YPKGJYDQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.549115876 (298h41m43.14588691s) [noise=true , address=145.79.227.46:4601 , piece_id=K7F57Y7XQ3VIUPN32DNLGY5H4NQHT3XIBLOJA5SR5OJPV6HQARQA, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.549528886 (298h41m43.146299916s) [noise=true , address=145.79.227.44:4198 , piece_id=SZN2PGVD24YOV3OCKCF3YUS7OSH2GZJ6U2S5SPCPHMNKCH4O3HJQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.549701152 (298h41m43.146472185s) [noise=true , address=145.79.227.29:4944 , piece_id=CLW5BZSW332GBJP3SHKVWM6RA2OBL2SOECNSK6K3GCMDHJBWTPIA, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.549974830 (298h41m43.146745859s) [noise=true , address=77.90.31.11:4557 , piece_id=7VO3YM6OR6EKGJOIIMKHX77F5PD2GTXC5FDGKFTGFE4FPTNWB7GA, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.550164567 (298h41m43.146935593s) [noise=true , address=45.91.116.9:4296 , piece_id=NCRYLRDGU4L3XPILHJ77P5TAP5TZQ36M6KXYBZEUVYGJGUXNH6UA, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.550443428 (298h41m43.147214457s) [noise=true , address=145.79.227.36:4518 , piece_id=75MYEQ3C7OCJTAKXYE63EKIUQLFVWQQCF2JDJESTG6IN3VOSYZ4Q, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.551057996 (298h41m43.147829019s) [noise=true , address=79.127.219.45:4854 , piece_id=LI4J4VKEA2G4AHESVANMUYYD6FMNFIBLUB2YGCTIN43EYWA7N6UQ, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.551660845 (298h41m43.148431891s) [noise=true , address=145.79.227.14:4841 , piece_id=GY6JKOC7BZSXW5GPQ4I4B64LMD3NJW2GLAY3WOPI76WCKT22QWMA, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.552142959 (298h41m43.148913982s) [noise=true , address=145.79.227.28:4725 , piece_id=JH2NOZXCEIAY5FYS6WTHAE5ZRJJSWTCX3FGTZ57KLOAITGBJVZ7A, seg_pos={0 0}, upload=334557]: Uploading piece... Mar 18 21:41:45.648177322 (298h41m43.244948345s) [upload=334557]: Next returned nil segment Mar 18 21:41:45.648190754 (298h41m43.24496178s) [upload=334557]: Waiting for error group managing segments... Mar 18 21:41:45.655272191 (298h41m43.252043233s) [noise=true , address=79.127.219.45:4854 , piece_id=LI4J4VKEA2G4AHESVANMUYYD6FMNFIBLUB2YGCTIN43EYWA7N6UQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.656277039 (298h41m43.253048123s) [noise=true , address=45.91.116.9:4296 , piece_id=NCRYLRDGU4L3XPILHJ77P5TAP5TZQ36M6KXYBZEUVYGJGUXNH6UA, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.656323624 (298h41m43.253094647s) [noise=true , address=145.79.227.46:4601 , piece_id=K7F57Y7XQ3VIUPN32DNLGY5H4NQHT3XIBLOJA5SR5OJPV6HQARQA, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.656983510 (298h41m43.253754533s) [noise=true , address=145.79.227.52:4448 , piece_id=MUXR27EU23SUFKVE5BRZPVLJZIZGNTTBOLANCCQ4FDJZZ2QJ55FQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.657400515 (298h41m43.254171547s) [noise=true , address=145.79.227.4:4190 , piece_id=MD5I5KWECLFX6WRJZZOZ7C5XGNNLJFCY7JN23ETN4ZZNBIYLRTXA, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.658163331 (298h41m43.254934355s) [noise=true , address=145.79.227.14:4841 , piece_id=GY6JKOC7BZSXW5GPQ4I4B64LMD3NJW2GLAY3WOPI76WCKT22QWMA, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.658777904 (298h41m43.255548928s) [noise=true , address=145.79.227.13:4807 , piece_id=64GOGTQX6HIMS4FEQ6EGDOK7GUUJHVFKJM7VODTFIMS6GRIGR73A, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.659398711 (298h41m43.256169724s) [noise=true , address=145.79.227.29:4944 , piece_id=CLW5BZSW332GBJP3SHKVWM6RA2OBL2SOECNSK6K3GCMDHJBWTPIA, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.659885179 (298h41m43.25665619s) [noise=true , address=145.79.227.28:4725 , piece_id=JH2NOZXCEIAY5FYS6WTHAE5ZRJJSWTCX3FGTZ57KLOAITGBJVZ7A, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.659993073 (298h41m43.2567641s) [noise=true , address=145.79.227.6:4263 , piece_id=IOYKTKNT3B65RLSEEQ7EABL4OZV5JNAPVE3ATOWVOT67AVVIZ3KQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.660423333 (298h41m43.257194346s) [noise=true , address=145.79.227.18:4011 , piece_id=PCVRQVDFHFHCD5FYQ54OCIBEIBWATMMFH6MRLIO2KXJ6YPKGJYDQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.660599454 (298h41m43.257370481s) [noise=true , address=145.79.227.26:4656 , piece_id=RU2GPQTGMSTCDP2SY4LFWANHCKPI2OIQ2DPHQ6K2N77K72SIH4HQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.661168890 (298h41m43.257939904s) [noise=true , address=145.79.227.31:4801 , piece_id=YDM3VXUVYWH6J2XNGQB6SWSQHV3XYJBNSQCHXINCTNGVDOZ6EUPQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: <nil> Mar 18 21:41:45.661180052 (298h41m43.257951066s) [seg_pos={0 0}, upload=334557]: Satisfied cohort requirements after 13 pieces Mar 18 21:41:45.661402386 (298h41m43.258173423s) [noise=true , address=145.79.227.36:4518 , piece_id=75MYEQ3C7OCJTAKXYE63EKIUQLFVWQQCF2JDJESTG6IN3VOSYZ4Q, seg_pos={0 0}, upload=334557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:45.661412591 (298h41m43.258183637s) [noise=true , address=145.79.227.36:4518 , piece_id=75MYEQ3C7OCJTAKXYE63EKIUQLFVWQQCF2JDJESTG6IN3VOSYZ4Q, seg_pos={0 0}, upload=334557]: Long tail cancellation due to optimal threshold Mar 18 21:41:45.661569233 (298h41m43.258340265s) [noise=true , address=145.79.227.44:4198 , piece_id=SZN2PGVD24YOV3OCKCF3YUS7OSH2GZJ6U2S5SPCPHMNKCH4O3HJQ, seg_pos={0 0}, upload=334557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:45.661584986 (298h41m43.258356023s) [noise=true , address=145.79.227.44:4198 , piece_id=SZN2PGVD24YOV3OCKCF3YUS7OSH2GZJ6U2S5SPCPHMNKCH4O3HJQ, seg_pos={0 0}, upload=334557]: Long tail cancellation due to optimal threshold Mar 18 21:41:45.661827248 (298h41m43.258598281s) [noise=true , address=77.90.31.11:4557 , piece_id=7VO3YM6OR6EKGJOIIMKHX77F5PD2GTXC5FDGKFTGFE4FPTNWB7GA, seg_pos={0 0}, upload=334557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:45.661838291 (298h41m43.258609327s) [noise=true , address=77.90.31.11:4557 , piece_id=7VO3YM6OR6EKGJOIIMKHX77F5PD2GTXC5FDGKFTGFE4FPTNWB7GA, seg_pos={0 0}, upload=334557]: Long tail cancellation due to optimal threshold Mar 18 21:41:45.661868637 (298h41m43.258639655s) [seg_pos={0 0}, upload=334557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:41:45.661987417 (298h41m43.258758431s) [upload=334557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:41:45.661995128 (298h41m43.258766152s) [upload=334557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:02.151448868 (298h41m59.748219902s) [upload=334557]: Done uploading segments... Mar 18 21:42:02.151472081 (298h41m59.748243122s) [upload=334557]: Upload finished. err: <nil> Duration: 30.003355281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X....X...XX] duration: 30.00170442s trace: https://jaeger.dev.storj.io/trace/4ce71f481dc738db FAIL (2026-03-18 21:41:58 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001731939s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 29.05987836s trace: https://jaeger.dev.storj.io/trace/3f03de6f2326ff28 FAIL (2026-03-18 21:41:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:35:56.276128004 (298h35m53.872899033s) [upload=334448]: Starting upload Mar 18 21:35:56.276257765 (298h35m53.873028791s) [upload=334448]: Done starting upload Mar 18 21:35:56.276325406 (298h35m53.873096419s) [upload=334448]: Uploading segments... Mar 18 21:35:56.276650904 (298h35m53.873421917s) [upload=334448]: Got next segment. Inline: true Mar 18 21:35:56.276659691 (298h35m53.873430706s) [upload=334448]: Waiting for error group managing segments... Mar 18 21:35:56.276753620 (298h35m53.873524644s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:35:56.276771520 (298h35m53.873542543s) [upload=334448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:35:56.276775297 (298h35m53.873546319s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:36:08.633522862 (298h36m6.23029391s) [upload=334448]: Done uploading segments... Mar 18 21:36:08.633550161 (298h36m6.230321188s) [upload=334448]: Upload finished. err: <nil> Duration:* 29.059905725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX.....X] duration: 30.006624173s trace: https://jaeger.dev.storj.io/trace/3f23889572a83afd FAIL (2026-03-18 21:41:48 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.006651413s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003508206s trace: https://jaeger.dev.storj.io/trace/ab8983060b5ff76 FAIL (2026-03-18 21:41:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:40:40.954811060 (298h40m38.551582084s) [upload=334541]: Starting upload Mar 18 21:40:40.954907056 (298h40m38.55167809s) [upload=334541]: Done starting upload Mar 18 21:40:40.955238030 (298h40m38.552009055s) [upload=334541]: Uploading segments... Mar 18 21:40:40.955468045 (298h40m38.55223907s) [upload=334541]: Got next segment. Inline: false Mar 18 21:40:40.955518690 (298h40m38.552289716s) [upload=334541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:40:40.955526754 (298h40m38.552297779s) [upload=334541]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:40:53.123698717 (298h40m50.720469732s) [seg_pos={0 0}, upload=334541]: Begin upload segment... Mar 18 21:40:53.123779064 (298h40m50.720550088s) [seg_pos={0 0}, upload=334541]: Done begin upload segment. Mar 18 21:40:53.123853631 (298h40m50.720624656s) [noise=true , address=145.79.227.42:4121 , piece_id=TVVL3U3ADVGARPPFYKIRGJ5GYZDJILUP5ACZS5GLAQFDO3YOYAVA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.127198862 (298h40m50.723969896s) [noise=true , address=145.79.227.36:4522 , piece_id=LH3I6HZ4TFEGDVR4WLBFS2XOF3SZMGKH6ORQ2TALIPEY2EQJK7JQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.127985147 (298h40m50.724756172s) [noise=true , address=45.91.116.5:4297 , piece_id=5WXSADOW3OXHKY4BZ4YWPE26V7IGDYFG2NRSNNFOINBTYRNQINPQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.128137392 (298h40m50.724908429s) [noise=true , address=145.79.227.15:4878 , piece_id=TRTIZ3NY35MK6DCTOIS3MSM3EURAJVTAEZYASGXLZEWTYKBBZKHQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.128698744 (298h40m50.725469767s) [noise=true , address=45.91.116.12:4255 , piece_id=RVWJ2S2BERSERSZ6AR63X4QFXQKADCOD3YH735SXS3EJLLGNWFQA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.128784480 (298h40m50.72555562s) [noise=true , address=145.79.227.51:4409 , piece_id=K4RNXIVNSJQ33JEGTSDWFLXZHJE3ERNHTAO5YPHZX57GL4WXCKRQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.129339802 (298h40m50.726110825s) [noise=true , address=145.79.227.7:4297 , piece_id=PWGGJNWQYGQXKR7RWZV2XUTKQM4TFOT4ME5MQV6QBLDWUJBJEOMA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.129443621 (298h40m50.726214651s) [noise=true , address=145.79.227.40:4049 , piece_id=SSCIK5G4IKOOZF25MSXNZ2VXIK3ZLQP6XRDM3V6LBIWW7AVSNU2Q, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.129889001 (298h40m50.726660023s) [noise=true , address=145.79.227.20:4443 , piece_id=YYHJCBM5U2L3HGQUZWKSEBEMSLVZHURB4CB7I2WHM5NAU7JPYAEA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.130062036 (298h40m50.726833069s) [noise=true , address=145.79.227.28:4728 , piece_id=KUZF24OB53CRVNXUANVFEQ3CBF4B4WCY7HEHELHSGJQ6SSSFCCZA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.130774845 (298h40m50.727545878s) [noise=true , address=145.79.227.33:4872 , piece_id=LTSWFWKY4KKQZPLENLA6JAXI7V3CJCEP4MD52EPRDPISVNYYM4PA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.132046760 (298h40m50.728817786s) [noise=true , address=77.90.31.19:4844 , piece_id=6JOLYVFBFBQB52E57DMMXE7MMCGOEOS5XBFHRLMJSH5C36PVKTVQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.132679293 (298h40m50.729450326s) [noise=true , address=145.79.227.45:4234 , piece_id=R2CONAMCO7XZTXYWFIZVCEFGBZ4K27AZT2PY3FF3AF4F474RBPUA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.133667683 (298h40m50.730438716s) [noise=true , address=145.79.227.32:4839 , piece_id=MW6GYSRRGN6U6GD6RQV5C6WU6IZHCA3S3Y4SPVXK3DYHIJ56WKLQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.134508860 (298h40m50.731279895s) [noise=true , address=145.79.227.25:4621 , piece_id=BN3U5CXXJHUNIFRS5L73COOR3CNXRY6U27NWFZAQY2GQJRDILOUQ, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.135265800 (298h40m50.732036827s) [noise=true , address=145.79.227.21:4477 , piece_id=VUYBMXKPHLKFNKZAF2BGWUXMPAGGMKCHN5KGEUW2NUNX2URHGDKA, seg_pos={0 0}, upload=334541]: Uploading piece... Mar 18 21:40:53.212049501 (298h40m50.808820528s) [upload=334541]: Next returned nil segment Mar 18 21:40:53.212067845 (298h40m50.808838875s) [upload=334541]: Waiting for error group managing segments... Mar 18 21:40:53.225073356 (298h40m50.821844387s) [noise=true , address=145.79.227.21:4477 , piece_id=VUYBMXKPHLKFNKZAF2BGWUXMPAGGMKCHN5KGEUW2NUNX2URHGDKA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.225808435 (298h40m50.822579466s) [noise=true , address=45.91.116.12:4255 , piece_id=RVWJ2S2BERSERSZ6AR63X4QFXQKADCOD3YH735SXS3EJLLGNWFQA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.226438663 (298h40m50.823209693s) [noise=true , address=145.79.227.7:4297 , piece_id=PWGGJNWQYGQXKR7RWZV2XUTKQM4TFOT4ME5MQV6QBLDWUJBJEOMA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.227992661 (298h40m50.824763693s) [noise=true , address=45.91.116.5:4297 , piece_id=5WXSADOW3OXHKY4BZ4YWPE26V7IGDYFG2NRSNNFOINBTYRNQINPQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.228530683 (298h40m50.825301704s) [noise=true , address=77.90.31.19:4844 , piece_id=6JOLYVFBFBQB52E57DMMXE7MMCGOEOS5XBFHRLMJSH5C36PVKTVQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.228903763 (298h40m50.825674797s) [noise=true , address=145.79.227.42:4121 , piece_id=TVVL3U3ADVGARPPFYKIRGJ5GYZDJILUP5ACZS5GLAQFDO3YOYAVA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.229404352 (298h40m50.826175382s) [noise=true , address=145.79.227.28:4728 , piece_id=KUZF24OB53CRVNXUANVFEQ3CBF4B4WCY7HEHELHSGJQ6SSSFCCZA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.229596003 (298h40m50.826367033s) [noise=true , address=145.79.227.15:4878 , piece_id=TRTIZ3NY35MK6DCTOIS3MSM3EURAJVTAEZYASGXLZEWTYKBBZKHQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.230037405 (298h40m50.826808432s) [noise=true , address=145.79.227.51:4409 , piece_id=K4RNXIVNSJQ33JEGTSDWFLXZHJE3ERNHTAO5YPHZX57GL4WXCKRQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.234369367 (298h40m50.831140397s) [noise=true , address=145.79.227.20:4443 , piece_id=YYHJCBM5U2L3HGQUZWKSEBEMSLVZHURB4CB7I2WHM5NAU7JPYAEA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.235266850 (298h40m50.832037885s) [noise=true , address=145.79.227.33:4872 , piece_id=LTSWFWKY4KKQZPLENLA6JAXI7V3CJCEP4MD52EPRDPISVNYYM4PA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.235919987 (298h40m50.832691014s) [noise=true , address=145.79.227.45:4234 , piece_id=R2CONAMCO7XZTXYWFIZVCEFGBZ4K27AZT2PY3FF3AF4F474RBPUA, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.236787993 (298h40m50.833559023s) [noise=true , address=145.79.227.32:4839 , piece_id=MW6GYSRRGN6U6GD6RQV5C6WU6IZHCA3S3Y4SPVXK3DYHIJ56WKLQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: <nil> Mar 18 21:40:53.236811012 (298h40m50.833582039s) [seg_pos={0 0}, upload=334541]: Satisfied cohort requirements after 13 pieces Mar 18 21:40:53.237079411 (298h40m50.833850448s) [noise=true , address=145.79.227.36:4522 , piece_id=LH3I6HZ4TFEGDVR4WLBFS2XOF3SZMGKH6ORQ2TALIPEY2EQJK7JQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr): protocol: expected piece hash; context canceled; EOF Mar 18 21:40:53.237090691 (298h40m50.833861719s) [noise=true , address=145.79.227.36:4522 , piece_id=LH3I6HZ4TFEGDVR4WLBFS2XOF3SZMGKH6ORQ2TALIPEY2EQJK7JQ, seg_pos={0 0}, upload=334541]: Long tail cancellation due to optimal threshold Mar 18 21:40:53.237297404 (298h40m50.834068438s) [noise=true , address=145.79.227.25:4621 , piece_id=BN3U5CXXJHUNIFRS5L73COOR3CNXRY6U27NWFZAQY2GQJRDILOUQ, seg_pos={0 0}, upload=334541]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS): protocol: expected piece hash; context canceled; EOF Mar 18 21:40:53.237306439 (298h40m50.834077471s) [noise=true , address=145.79.227.25:4621 , piece_id=BN3U5CXXJHUNIFRS5L73COOR3CNXRY6U27NWFZAQY2GQJRDILOUQ, seg_pos={0 0}, upload=334541]: Long tail cancellation due to optimal threshold Mar 18 21:40:53.237437801 (298h40m50.834208835s) [noise=true , address=145.79.227.40:4049 , piece_id=SSCIK5G4IKOOZF25MSXNZ2VXIK3ZLQP6XRDM3V6LBIWW7AVSNU2Q, seg_pos={0 0}, upload=334541]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Mar 18 21:40:53.237452906 (298h40m50.834224039s) [noise=true , address=145.79.227.40:4049 , piece_id=SSCIK5G4IKOOZF25MSXNZ2VXIK3ZLQP6XRDM3V6LBIWW7AVSNU2Q, seg_pos={0 0}, upload=334541]: Long tail cancellation due to optimal threshold Mar 18 21:40:53.237483225 (298h40m50.834254238s) [seg_pos={0 0}, upload=334541]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:40:53.237576309 (298h40m50.834347337s) [upload=334541]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:40:53.237583188 (298h40m50.834354212s) [upload=334541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:41:09.834989661 (298h41m7.431760684s) [upload=334541]: Done uploading segments... Mar 18 21:41:09.835059565 (298h41m7.43183059s) [upload=334541]: Upload finished. err: <nil> Duration: 30.003625659s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXX] duration: 5.001287055s trace: https://jaeger.dev.storj.io/trace/60e9c453ba62ed71 FAIL (2026-03-18 21:41:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001313357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X....X...X] duration: 30.001545329s trace: https://jaeger.dev.storj.io/trace/5b4b3b3b30a012b1 FAIL (2026-03-18 21:40:47 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001575409s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 29.254834772s trace: https://jaeger.dev.storj.io/trace/3f03de6f2326ff28 FAIL (2026-03-18 21:40:46 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:35:56.276128004 (298h35m53.872899033s) [upload=334448]: Starting upload Mar 18 21:35:56.276257765 (298h35m53.873028791s) [upload=334448]: Done starting upload Mar 18 21:35:56.276325406 (298h35m53.873096419s) [upload=334448]: Uploading segments... Mar 18 21:35:56.276650904 (298h35m53.873421917s) [upload=334448]: Got next segment. Inline: true Mar 18 21:35:56.276659691 (298h35m53.873430706s) [upload=334448]: Waiting for error group managing segments... Mar 18 21:35:56.276753620 (298h35m53.873524644s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:35:56.276771520 (298h35m53.873542543s) [upload=334448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:35:56.276775297 (298h35m53.873546319s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:36:08.633522862 (298h36m6.23029391s) [upload=334448]: Done uploading segments... Mar 18 21:36:08.633550161 (298h36m6.230321188s) [upload=334448]: Upload finished. err: <nil> Duration:* 29.254866854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XX] duration: 5.00189229s trace: https://jaeger.dev.storj.io/trace/4c17b282394c3717 FAIL (2026-03-18 21:40:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001913467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 26.329476526s trace: https://jaeger.dev.storj.io/trace/6719b71e4a246259 FAIL (2026-03-18 21:40:30 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.329513042s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 18.813455668s trace: https://jaeger.dev.storj.io/trace/3f03de6f2326ff28 FAIL (2026-03-18 21:40:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:35:56.276128004 (298h35m53.872899033s) [upload=334448]: Starting upload Mar 18 21:35:56.276257765 (298h35m53.873028791s) [upload=334448]: Done starting upload Mar 18 21:35:56.276325406 (298h35m53.873096419s) [upload=334448]: Uploading segments... Mar 18 21:35:56.276650904 (298h35m53.873421917s) [upload=334448]: Got next segment. Inline: true Mar 18 21:35:56.276659691 (298h35m53.873430706s) [upload=334448]: Waiting for error group managing segments... Mar 18 21:35:56.276753620 (298h35m53.873524644s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:35:56.276771520 (298h35m53.873542543s) [upload=334448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:35:56.276775297 (298h35m53.873546319s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:36:08.633522862 (298h36m6.23029391s) [upload=334448]: Done uploading segments... Mar 18 21:36:08.633550161 (298h36m6.230321188s) [upload=334448]: Upload finished. err: <nil> Duration:* 18.813493063s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..X] duration: 5.003247051s trace: https://jaeger.dev.storj.io/trace/46ab2f3664764a8f FAIL (2026-03-18 21:40:03 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003280655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 14.056728027s trace: https://jaeger.dev.storj.io/trace/6719b71e4a246259 FAIL (2026-03-18 21:39:47 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 14.056762704s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X....X...] duration: 243.939545ms trace: https://jaeger.dev.storj.io/trace/1cdd5dfb0a0140ba FAIL (2026-03-18 21:39:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 243.974439ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..] duration: 1.058569341s trace: https://jaeger.dev.storj.io/trace/7aca8a6616cd44bd FAIL (2026-03-18 21:38:44 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.058882518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 402.566377ms trace: https://jaeger.dev.storj.io/trace/3f03de6f2326ff28 FAIL (2026-03-18 21:38:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:35:56.276128004 (298h35m53.872899033s) [upload=334448]: Starting upload Mar 18 21:35:56.276257765 (298h35m53.873028791s) [upload=334448]: Done starting upload Mar 18 21:35:56.276325406 (298h35m53.873096419s) [upload=334448]: Uploading segments... Mar 18 21:35:56.276650904 (298h35m53.873421917s) [upload=334448]: Got next segment. Inline: true Mar 18 21:35:56.276659691 (298h35m53.873430706s) [upload=334448]: Waiting for error group managing segments... Mar 18 21:35:56.276753620 (298h35m53.873524644s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:35:56.276771520 (298h35m53.873542543s) [upload=334448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:35:56.276775297 (298h35m53.873546319s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:36:08.633522862 (298h36m6.23029391s) [upload=334448]: Done uploading segments... Mar 18 21:36:08.633550161 (298h36m6.230321188s) [upload=334448]: Upload finished. err: <nil> Duration:* 402.597545ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X....X..] duration: 365.081258ms trace: https://jaeger.dev.storj.io/trace/1cdd5dfb0a0140ba FAIL (2026-03-18 21:38:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 365.132151ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX.] duration: 3.066564061s trace: https://jaeger.dev.storj.io/trace/7aca8a6616cd44bd FAIL (2026-03-18 21:38:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.066656892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.739931889s trace: https://jaeger.dev.storj.io/trace/6719b71e4a246259 FAIL (2026-03-18 21:38:29 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.739984706s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX] duration: 5.00158104s trace: https://jaeger.dev.storj.io/trace/7aca8a6616cd44bd FAIL (2026-03-18 21:38:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001605214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 1.958222233s trace: https://jaeger.dev.storj.io/trace/6719b71e4a246259 FAIL (2026-03-18 21:38:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.958309588s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 19.534680032s trace: https://jaeger.dev.storj.io/trace/6719b71e4a246259 FAIL (2026-03-18 21:38:07 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 19.534723427s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 25.830578284s trace: https://jaeger.dev.storj.io/trace/1cdd5dfb0a0140ba FAIL (2026-03-18 21:38:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 25.830599487s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 28.641667191s trace: https://jaeger.dev.storj.io/trace/3f03de6f2326ff28 FAIL (2026-03-18 21:38:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:35:56.276128004 (298h35m53.872899033s) [upload=334448]: Starting upload Mar 18 21:35:56.276257765 (298h35m53.873028791s) [upload=334448]: Done starting upload Mar 18 21:35:56.276325406 (298h35m53.873096419s) [upload=334448]: Uploading segments... Mar 18 21:35:56.276650904 (298h35m53.873421917s) [upload=334448]: Got next segment. Inline: true Mar 18 21:35:56.276659691 (298h35m53.873430706s) [upload=334448]: Waiting for error group managing segments... Mar 18 21:35:56.276753620 (298h35m53.873524644s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:35:56.276771520 (298h35m53.873542543s) [upload=334448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:35:56.276775297 (298h35m53.873546319s) [upload=334448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:36:08.633522862 (298h36m6.23029391s) [upload=334448]: Done uploading segments... Mar 18 21:36:08.633550161 (298h36m6.230321188s) [upload=334448]: Upload finished. err: <nil> Duration:* 28.641695952s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)